node, aggregation: zeam aggregate-build is 5–7x slower than ethlambda on devnet (2–3s vs 0.4s)

[ch4r10t33r]

Summary

On the multi-client devnet (8 aggregators across zeam / ethlambda / ream), zeam's per-aggregate build time is 5–7× higher than ethlambda's, measured cross-client at the same wall-clock with the same standard lean_pq_sig_aggregated_signatures_building_time_seconds histogram.

Aggregator	mean build time	sample count	head_slot
zeam_8	2.90 s	64	256
zeam_4	2.91 s	48	227
zeam_16	2.00 s	64	261
ethlambda_8	0.43 s	88	271
ethlambda_5	0.41 s	8	108
ethlambda_16	0.40 s	12	263
ream_0	(metric not exposed — see Notes)	—	223
ream_13	(metric not exposed)	—	215

(All numbers from each host's local Prometheus, snapshot 2026-05-19T15:13Z, ~30 min after fresh devnet restart.)

Verify side is fine across all clients (aggregated_signatures_verification_time is 35–71 ms; attestation_verification_time is 0.78–1.25 ms). The cost gap is squarely on the build path.

drops and queue saturation are clean on this run (lean_chain_queue_dropped_total = 0 for block and attestation queues on every zeam aggregator), so the slowness is not a queue-backpressure artifact and is not the regression PR #894 fixed. It's intrinsic per-aggregate cost.

Why it matters — direct FFG impact

Slow build → zeam aggregators frequently miss a slot's aggregation window (zeam_aggregate_skip_total{reason="in_flight"} ticks up because aggregate_group.concurrent is gated at concurrent_limit=1, see pkgs/node/src/chain.zig:4357). The downstream effect is visible directly in the next block proposal's coverage. From zeam_8 log at slot 256:

[forkchoice] block proposal slot=256: attestation aggregate coverage:
  payloads = subnet0=3/8(37.50%)  subnet1=8/8(100%)  subnet2=7/8(87.50%)
             subnet3=8/8(100%)   subnet4=7/8(87.50%) subnet5=1/8(12.50%)
             subnet6=6/8(75%)    subnet7=6/8(75%)
  network=46/64(71.88%)

Subnets 0, 3, 5 are the three zeam-aggregator subnets. Subnet 5 contributed 1 of 8 votes. Subnet 0 contributed 3 of 8. Network coverage hovers around 70–72%, marginal vs the 67% justification threshold once any vote is on a divergent target — currently latest_justified_slot=0 and latest_finalized_slot=0 for every node in the devnet.

So this issue is the dominant single contributor to "no FFG progress" on the current cross-client devnet (other contributors — #863 slot-driver backlog, head divergence — are already tracked).

Where the cost lives

• Entry: chain.zig submitAggregateOnInterval → aggregate_group.concurrent on the dedicated aggregate_io thread (the move done in Off aggregate from libxev thread to avoid slot advance blocking #873).
• Worker: chain.zig aggregateImpl (pkgs/node/src/chain.zig:4371) →
  forkChoice.aggregate → aggregateUnlocked (pkgs/node/src/forkchoice.zig:2035).

aggregateUnlocked is structured in three phases:

1. Snapshot under signatures_mutex — AggregateSnapshot.takeUnderLock. Comment claims O(10 ms) typical.
2. Compute with no lock held — agg.computeAggregatedSignatures(&state.validators, &snap.signatures, &snap.new_payloads, &snap.known_payloads). This is the heavy XMSS FFI path. The same docstring on aggregate() at line 2240 says "The ~18 s computeAggregatedSignatures runs in between with no locks" — i.e. the worst case acknowledged in code is ~18 s, and the live mean we measure today is 2–3 s.
3. Commit under signatures_mutex — merge results, prune consumed snapshot vids. ms-scale.

So the work that's slow is phase 2 only — the FFI into multisig-glue for XMSS aggregation/proof-construction. Phases 1 and 3 are not the cost drivers.

ethlambda doing the equivalent work in 0.4 s is the strongest signal that this is not a fundamental cost of the spec — there is something specific to zeam's multisig FFI path that is 5–7× off.

Notes on cross-client comparability

• lean_pq_sig_aggregated_signatures_building_time_seconds is part of the standard leanMetrics set, so the histogram name and semantics match across zeam / ethlambda by construction.
• ream does not expose this histogram (or _total, _invalid_total, _in_aggregated_signatures_total, or the individual attestation_verification_time histogram). Their /metrics only carries _valid_total + _verification_time_seconds for aggregated sigs. That very likely means ream does not generate a recursive/SNARK aggregation proof at all (just packs verified XMSS sigs + a participation bitfield), and therefore has nothing to time. Worth confirming with the ream team before drawing conclusions, but it does suggest that even a SNARK-free packing is spec-acceptable.

Suggested investigation

1. Profile multisig-glue aggregation FFI under a representative load (8 raw atts, validator set 64) on a single host. Identify whether the cost is in:
   • per-signature recursive-proof witness construction (parallelisable per signature)
   • shared setup / SRS load (cacheable, currently re-done?)
   • bigint / hash-chain work (constant factor)
   • any debug-mode build artefact in the rust crate that landed on the devnet image.
2. Compare the multisig-glue API surface with what ethlambda uses. If they use a different (lighter) aggregation construction, decide whether to align — or document the cost/security trade-off.
3. Check parallelism. aggregate_io is concurrent_limit=1 (Off aggregate from libxev thread to avoid slot advance blocking #873), so two slots' aggregations cannot run in parallel, but within a single aggregation the per-signature work could be parallelised across a thread pool. Currently computeAggregatedSignatures appears to call FFI sequentially per attestation; verify and parallelise if so.
4. Verify Rust release-mode / opt-level on the devnet build. A 5–7× hit is consistent with debug-mode FFI, so rule that out first.
5. Add per-phase timing to aggregateUnlocked — break the histogram into phase1_snapshot, phase2_compute_ffi, phase3_commit so we can tell from Prometheus alone whether the cost is in the FFI or in surrounding Zig work.

Acceptance criteria

• Mean lean_pq_sig_aggregated_signatures_building_time_seconds on a zeam aggregator ≤ 0.6 s on the multi-client devnet (matching ethlambda within ~1.5×).
• On a 32-validator-per-subnet stress run, p99 build time ≤ 2 s.
• zeam_aggregate_skip_total{reason="in_flight"} rate on a zeam aggregator drops by ≥ 4× compared to today's devnet baseline.
• Per-phase build histogram exposed (phase1_snapshot, phase2_compute_ffi, phase3_commit) so we can attribute future regressions.
• Block-proposal aggregate coverage from a zeam-managed subnet (0, 3, 5) reaches ≥ 6/8 in steady state on the same devnet.

Related

• Off aggregate from libxev thread to avoid slot advance blocking #873 — (closed) off-load aggregate from libxev thread. This issue is downstream of that — even on the dedicated aggregate_io thread, the per-aggregate cost itself is 5–7× ethlambda.
• node: parallel XMSS verify with serial fork-choice apply for gossip aggregations #895 — parallel verify with serial fork-choice apply for incoming gossip aggregations. Symmetric concern on the verify path; this issue is the produce path.
• Investigate slow slot_interval / tick duration (event-loop starvation vs nominal 0.8s) #863 — slow slot_interval / libxev event-loop starvation. Slow aggregate build is one of the largest contributors to the kind of long FFI completions Investigate slow slot_interval / tick duration (event-loop starvation vs nominal 0.8s) #863 is investigating, even now that build has been moved off the libxev thread.
• node: aggregator-only freeze of slot-tick / onInterval loop after MissingPreState cascade (separate from #837 STF reject) #847 — aggregator-only freeze after MissingPreState cascade. Different defect, same surface (aggregator behaviour under load).

Devnet repro context

• Branch under test on zeam aggregators: PR node: harden blocks_by_range catch-up retry and fork recovery #894 (commit b2654679).
• Date: 2026-05-19, devnet restarted at ~14:43 UTC.
• Topology: 24 zeam + 24 ethlambda + 16 ream validators across 8 subnets (gean and nlean slots replaced with zeam/ethlambda in lean-quickstart ansible-devnet/genesis/validator-config.yaml).
• Live at snapshot time: head spread 108–271, jst=0, fin=0 on every aggregator.

[ch4r10t33r]

Follow-up measurements — devnet restart 2026-05-19 evening (~22:30 UTC)

Second fresh-genesis restart on the same 8-subnet ansible devnet (24 zeam + 24 ethlambda + 16 ream). Snapshots at T+10 min and T+15 min after containers reached head>0.

lean_pq_sig_aggregated_signatures_building_time_seconds (mean)

Subnet	Aggregator	head (T+15m)	agg built	build_avg	verify_avg	votes/agg
0	zeam_8	149	32	4.57 s	45 ms	~23
1	ethlambda_8	149	230	1.15 s	75 ms	~8
2	ream_0	97	—	(not exposed)	78 ms	—
3	zeam_4	149	32	4.30 s	50 ms	~16
4	ethlambda_5	151	233	1.23 s	75 ms	~9
5	zeam_16	149	37	4.76 s	47 ms	~24
6	ream_13	101	—	(not exposed)	77 ms	—
7	ethlambda_16	151	242	1.23 s	76 ms	~8

Comparison to first snapshot (same day, ~15:13 UTC)

Client	build_avg (1st run)	build_avg (2nd run)	Change
zeam (3 aggs)	2.0 – 2.9 s	4.3 – 4.8 s	~1.6× slower
ethlambda (3 aggs)	0.40 – 0.43 s	1.15 – 1.23 s	~3× slower
ream	not exposed	not exposed	unchanged

The zeam vs ethlambda gap persists (~4×) and both clients regressed vs the afternoon session. Verify time remains cheap (45–78 ms) on all clients — still a build-path-only problem.

Other observations (2nd run)

• Head spread: zeam + ethlambda aligned at 149–151; ream lagging at 97–101 (~50 slots behind).
• FFG: still justified=0 / finalized=0 on all aggregators at T+15 min.
• PR Limit aggregate signature builds to active slot #900 (limit aggregate builds to active slot) merged — does not close this issue; it addresses in-flight/slot-window behaviour, not the per-build FFI cost measured here.
• Queue drops: not re-checked this pass; afternoon run had lean_chain_queue_dropped_total = 0 on zeam aggregators.

Conclusion

Issue remains valid and open. The multisig-glue build cost investigation and acceptance criteria below still apply. Next step: confirm deployed image tags on this restart vs the afternoon run (cold-cache vs image drift).

[zclawz]

Quick code-read on the zeam side: I think the most likely cause is not a debug build, but zeam's current Rayon pool sizing for leanMultisig.

Evidence:

• Docker/release path does build Rust in release mode:

  • Dockerfile runs zig build -Doptimize=ReleaseSafe ...
  • build.zig invokes cargo build --release -p zeam-glue --features libp2p,hashsig,multisig
  • amd64 images pass -Drust-target-cpu=x86-64-v3, and build.zig turns that into RUSTFLAGS=-Ctarget-cpu=x86-64-v3 -Dwarnings, so AVX2 cfgs in leanMultisig's backend should be enabled.

• But at runtime zeam explicitly shrinks Rayon before leanMultisig is initialized (pkgs/cli/src/node.zig):

  const cpu_count = std.Thread.getCpuCount() catch 2;
  const reserved_system_threads: usize = 4;
  const desired_workers = @max(@as(usize, 1), cpu_count -| reserved_system_threads);
  const zig_worker_budget = @max(@as(usize, 1), (desired_workers + 1) / 2);
  const worker_count = @min(zig_worker_budget, @as(usize, ThreadPool.max_thread_count));
  ...
  const rayon_threads = @max(@as(usize, 1), desired_workers -| worker_count);
  xmss.setRayonThreads(rayon_threads);

  On typical devnet-sized containers this can leave leanMultisig with only 1–2 Rayon threads:

  • 4 vCPU: desired_workers=1, worker_count=1, rayon_threads=1
  • 8 vCPU: desired_workers=4, worker_count=2, rayon_threads=2

  leanMultisig's prover is heavily Rayon-parallel (prove_execution, WHIR/DFT/sumcheck/Merkle code paths use par_iter, par_chunks, etc.). Ethlambda appears not to cap Rayon; it builds with -Ctarget-cpu=x86-64-v3 in .cargo/config.toml and lets Rayon default to logical CPU count. A 1–2 thread zeam prover vs a 4–8 thread ethlambda prover is very plausibly the measured ~4–7× gap.

• Build optimisation still has one possible gap worth benchmarking: zeam's Rust workspace overrides release profile with lto = false and codegen-units = 16, while upstream leanMultisig's own workspace has [profile.release] lto = "thin". Cargo uses the top-level workspace profile, so zeam is likely compiling leanMultisig without upstream ThinLTO. That may be a constant-factor hit, but I would check Rayon thread count first because it lines up better with the observed multiplier.

Suggested next checks:

1. Log/export cpu_count, worker_count, and rayon_threads at node startup for each aggregator image.
2. Run one local/devnet A/B with the same image but force Rayon higher (e.g. all remaining CPUs, or env/build option if added) and compare lean_pq_sig_aggregated_signatures_building_time_seconds.
3. Benchmark Rust profile variants separately: current (lto=false, codegen-units=16) vs lto="thin", codegen-units=1 for dummy/single-prover builds.
4. If the Rayon hypothesis holds, change the production aggregator policy to give leanMultisig most cores during aggregate build, while keeping verification/network work bounded. The current split is conservative for general node responsiveness but bad for a latency-critical prover.

So: we are probably not missing release/AVX2 build flags, but we may be missing the most important leanMultisig optimisation in practice: enough Rayon parallelism. ThinLTO is the second suspect.

[ch4r10t33r]

Follow-up measurements — devnet restart 2026-05-26 (~11:14 UTC)

Third measurement pass on the ansible devnet after deploying 0xpartha/zeam:local with merged #925 (aggregate slot-budget / Rayon auto-tune) and #933 (aggregator commit segfault fix). Snapshot at T+2 h (head≈2000, container RestartCount=0).

lean_pq_sig_aggregated_signatures_building_time_seconds (mean)

Aggregator	head	agg built	build_avg	verify_avg (cross-check)
zeam_8	2005	169	1.34 s	45–50 ms
ethlambda_8	2025	3705	1.42 s	~62 ms

zeam vs ethlambda gap is now ~1.06× (parity), down from the original 4–7× gap and the May-19 evening ~4× gap.

Histogram shape on zeam_8 (n=169): p50 ≈ 0.5 s (91/169 ≤ 0.5 s), p95 ≈ 4 s (161/169 ≤ 4 s). Mean is pulled up by a long tail of multi-attestation builds.

Comparison across measurement passes

Pass	zeam build_avg	ethlambda build_avg	zeam/ethlambda
May-19 ~15:13 UTC (1st)	2.0 – 2.9 s	0.40 – 0.43 s	~5–7×
May-19 ~22:30 UTC (2nd)	4.3 – 4.8 s	1.15 – 1.23 s	~4×
May-26 ~13:05 UTC (3rd)	1.34 s	1.42 s	~1×

Both clients regressed vs the original afternoon ethlambda baseline (0.4 s); the cross-client gap on the current devnet is no longer zeam-specific.

Where the cost still lives

zeam_xmss_rec_aggregate_prove_seconds tracks the build histogram almost exactly on zeam_8 (226.33 s / 169 vs 226.78 s / 169) — phase-2 leanMultisig STARK prove remains the dominant cost. New metrics from #925:

• zeam_aggregate_worker_duration_seconds: 229.96 s / 1690 runs → ~0.14 s mean per worker tick; 1634/1690 ticks finish in ≤ 10 ms (no new payloads). Heavy runs align with the build histogram tail.
• zeam_node_aggregation_interval_tick_seconds: ~40 µs (libxev submit path is not the bottleneck).

Slot-window / in-flight behaviour (#900 + #925 coalesce)

• zeam_aggregate_skip_total: no increments on this run (in-flight triggers coalesce instead of skip).
• zeam_aggregate_coalesced_total: 53 — worker catch-up after in-flight build.
• Publish lag (last 20 published aggregation log lines on zeam_8): {0: 2, 1: 14, 2: 4} slots, mean 1.1 slots (was ~2+ slots pre-node, types, xmss: keep aggregate worker under 4s slot budget (#907) #925 in earlier monitoring).

Rayon thread budget (@zclawz hypothesis)

zeam_8 startup log (16 vCPU container):

startup thread budget: cpu_count=16 estimated_os_threads≈22
  rayon (XMSS prover):  12 (aggregator auto-tune: full post-system budget)

This is the #925 aggregator policy — leanMultisig gets the full post-system CPU budget instead of the previous 1–2 Rayon threads. Strongly correlates with the ~3× build-time drop vs the May-19 evening run.

Downstream FFG / coverage impact

Coverage (zeam proposer slots 1536–1984, steady state):

subnet0=8/8  subnet3=8/8  subnet5=8/8  (zeam-managed subnets)
network=49/64 (76.56%) — subnet2 (ream) weak at 2/8

Meets the ≥ 6/8 zeam-subnet coverage acceptance criterion. Network coverage above the 67% justification threshold when votes align.

FFG: justified=25, finalized=0 on zeam_8 at snapshot (progress vs jst=0/fin=0 on all nodes in prior passes). Finalization still blocked — likely separate from build latency at this point.

Acceptance criteria check

Criterion	Status
Mean build ≤ 0.6 s (match ethlambda within ~1.5×)	Partial — p50 ≈ 0.5 s but mean 1.34 s; ethlambda mean also 1.42 s on current devnet
p99 build ≤ 2 s (32-validator stress)	Likely OK on current 8-v/subnet topology; tail hits 2–4 s on multi-att builds
zeam_aggregate_skip_total{reason="in_flight"} rate ↓ ≥ 4×	Improved — skips replaced by coalesce (#925); coalesced=53 over ~2000 slots
Per-phase build histogram	Partial — zeam_xmss_rec_aggregate_prove_seconds + zeam_aggregate_worker_duration_seconds added; still no phase1_snapshot / phase3_commit split inside aggregateUnlocked
Zeam-managed subnet coverage ≥ 6/8 steady state	Met — 8/8 on subnets 0, 3, 5

Conclusion

The original zeam-specific 5–7× multisig build regression is resolved on the current image (#925 Rayon auto-tune + coalesce path). Remaining work to fully close this issue:

1. Drive mean build toward ≤ 0.6 s — tail of multi-attestation builds (2–4 s) dominates mean; investigate parallel per-signature prove within a single aggregate and/or ThinLTO profile (@zclawz item 3).
2. Expose phase1/phase3 timing in Prometheus for future regression attribution.
3. Track ethlambda regression (0.4 s → 1.4 s) — may be devnet load / vote volume growth rather than client-specific; worth a controlled A/B with fixed attestation count.

Related: gossip stall / catch-up recovery tracked separately in #926 (PR #938).

[zclawz]

Thanks for the detailed follow-up, @ch4r10t33r! Great to see the zeam-specific gap closed — the Rayon auto-tune (#925) lining up with the hypothesis is satisfying.

For the remaining open items I can track:

1. Drive mean ≤ 0.6 s — the multi-attestation tail (2–4 s) is the blocker; will look at whether parallelising per-signature prove within a single aggregate call is feasible in the current multisig-glue API, and whether ThinLTO in the Rust profile helps.
2. Phase1/phase3 histogram — can add phase1_snapshot_seconds and phase3_commit_seconds Prometheus observations inside aggregateUnlocked straightforwardly.
3. ethlambda regression (0.4 → 1.4 s) — probably vote-volume growth with the larger devnet; worth a controlled A/B before attributing to client changes.

Will open follow-up PRs for 1 and 2 once I have had a look at the multisig-glue surface.