Simulation results for tag leios-2025w09

[bwbush]

PR #213 contains simulation scripts and results for leios-2025w09.

The Jupyter notebook exploration.ipynb presents the analysis results.

[bwbush]

First experiment

Scenario definition

The first experiment does the following:

• Defaults for protocol configuration: config.defaults.yaml
• 100-node network: topo-default-100.yaml
• IB production rate varies from 0.1 to 100 IB/s
• IB size is 102400 bytes
  • The Haskell simulation handles this naturally
  • I had to set the IB header to 102704 bytes to mimic this
• Transactions are turned off

Results

✔️ Both simulations generate IBs at the proper rate.

✔️ Both simulations generate blocks of 103 kB.

✖️ The Haskell simulation shows a larger spread for in-flight time for blocks.

Haskell	Rust

✖️ Congestion occurs at ~ 40 IB/s in the Haskell but at ~ 15 IB/s in the Rust. The distributions of fraction of IBs received as a function of time are quite different: breakdown is rapid and constant in Rust but it is more gradual and increasing in Haskell.

Haskell	Rust

[SupernaviX]

What does "transactions are turned off" mean in that description? Rust doesn't directly support that right now, and the "default config" looks like it's producing ~1 TX per second.

[bwbush]

Yes, a few transactions slip through, but I set the following values in the config file:

• tx-generation-distribution: {distribution: "constant", value: 1000000}
• ib-head-size-bytes: 102704

Without modifying the source code, this was the best I could do to turn off transactions but set the IBs to a fixed size.

[pagio]

An interesting metric here is the data throughput to available bandwidth relation.

• Throughput can be calculated as (IB rate) * (IB size) ~ 40 * 0.1MBps ~ 4MBps
• The available bandwidth per node is (bandwidth per connection)*(#connections) ~ 1MBps * 12 ~ 12MBps. For the number of connections here I used the number of connections of node 0. Ideally, we should use the minimum #connections found in https://github.com/input-output-hk/ouroboros-leios/blob/bwbush/leios-2025w09/analysis/sims/2025w09/topo-default-100.yaml.

Hence, we get that throughput is ~ 1/3 of the available bandwidth nodes have. It would be interesting to compare throughput to the average bandwidth used as well.

As I mentioned in slack, modelling bandwidth independently in each channel may be somewhat unrealistic, as I would expect a total bandwidth constraint per node.

[bwbush]

@SupernaviX, I'm trying to understand why the in-flight time varies so much on a lightly-loaded network in the Rust simulation.

• 0.5 IB/s
• 100 kB/IB
• all CPU-usage parameters set to 0 ms

For example, on the link from node 63 to node 11, I see three IBs transmitted, but the one (IB 26-node-40-0) at 27 seconds takes a lot longer to traverse the link than the other two. All of the blocks are the same size, 100304 bytes (including the header).

Node 63 is sending the block to three downstream nodes near this time, but the link to node 11 is only used once. Why does is the flight time longer?

[SupernaviX]

Links are modeled in src/network/connection.rs. I linked to the version of that file used when this simulation ran.

On Friday, all messages were transmitted simultaneously, and the bandwidth was divvied among the messages. This is incorrect, because within a miniprotocol messages are sent one-at-a-time and arrive in FIFO order.

On Monday, I updated it so that one message is sent at a time (or rather, one message gets all the bandwidth at any given time; if latency is 50ms, you don't have to wait 50ms between messages). This is also incorrect, because different kinds of messages are sent over different miniprotocols, so links shouldn't be totally FIFO.

My next update will be to make this "miniprotocol-aware", and divide bandwidth up "evenly" across miniprotocols the way that the real implementation does.

[SupernaviX]

I verified that there are no transactions, EBs, or votes at either end of the link during this time period.

In that case, I'm confused where any discrepancy could come from. There is network traffic not accounted for by the exposed events; every resource uses "announce", "request" and "send" messages, and the event log only includes "send" events. But for bandwidth purposes, that traffic should be negligible; every "announce" and "request" message is just 8 bytes.

[SupernaviX]

Does the Rust simulator model contention at the network interface card (NIC) or between the machine and the first switch? or is there no contention for NIC resources?

Forgive my ignorance, but I don't know what the difference between these would be in practice.

The rust simulation delays a message by some amount of time due to bandwidth (contention makes this take longer), plus some amount of time based on latency. Bandwidth is a property of an individual link, and applied separately for messages going either direction on that link (i.e. we track bandwidth separately for A->B and B->A). The topology says every link has 1 MB/s bandwidth, so every node can simultaneously transmit 1 MB/s to every other node (ignoring latency).

[bwbush]

I created a detailed transcript of that link and see that the congestion was caused by headers being fetched. The log file doesn't report headers, so I didn't see that network activity until I added extra logging. Anyway, the mystery is solved.

I did also notice that there is some ping-pong of headers back upstream and duplicate announcements. For example:

# Upstream announces slot 26 header of node 82
FORWARD Timestamp(26.825670968s) AnnounceIBHeader(InputBlockId { slot: 26, producer: NodeId(82), index: 0 })

# Downstream requests slot 26 header of node 82
REVERSE Timestamp(26.864219963s) RequestIBHeader(InputBlockId { slot: 26, producer: NodeId(82), index: 0 })

# Upstream sends slot 26 header of node 82
FORWARD Timestamp(26.902768958s) IBHeader(InputBlockHeader { id: InputBlockId { slot: 26, producer: NodeId(82), index: 0 }, vrf: 21, timestamp: Timestamp(26s), bytes: 100304 }, false)

# Downstream announces slot 26 header of node 82 even though that's from where it received the header
REVERSE Timestamp(27.909401926s) AnnounceIB(InputBlockId { slot: 26, producer: NodeId(82), index: 0 })

[SupernaviX]

Downstream is announcing the body to upstream. It received the header from upstream, but that doesn't necessarily mean that upstream has the body.

[Saizan]

The Haskell simulation shows a larger spread for in-flight time for blocks.

@SupernaviX how does the rust simulation interleave validation with relay? e.g. In the haskell sim each node goes through these steps, in order:

1. header received
2. header validated
3. body requested
4. body received
5. wait for ledger state of referenced RB
6. body validated
7. block can now be announced.
   The ledger state for an RB requires having validated all the IBs and EBs referenced by the chain up to and including the RB.

If the Rust sim is e.g. announcing the header already after 2 it could explain some of the smaller spread.

[SupernaviX]

@Saizan the rust sim is announcing the IB header after it's validated. Why wait until you have the body?