node: parallel XMSS verify with serial fork-choice apply for gossip aggregations

[ch4r10t33r]

Context

Fresh devnet restarts can flood the node with gossip aggregated attestations while blocks_by_range catch-up keeps the chain-worker busy on block STF. The aggregated-attestation queue (512) then fills faster than a single worker drains it, producing aggregated attestation queue full, dropping slot=… and starving fork-choice vote tracking.

PR #894 mitigates this with:

• buffer-on-QueueFull → pending_aggregated_attestations (leanSpec-style)
• batch drain (up to 32 aggs per loop when depth > 16)
• less per-agg work (single participant-bit decode)

Those are backpressure / throughput tweaks on the existing architecture: one chain-worker thread applies blocks, raw attestations, and aggregations in a fixed loop (block first, then one att, then one-or-many aggs).

Problem

One applier thread is correct for consistency (fork choice, states map, historical roots must stay ordered), but it becomes the bottleneck when verify + apply are colocated on that thread under gossip burst + block catch-up.

We already parallelize some work elsewhere:

• local aggregation production: dedicated aggregate_io (Off aggregate from libxev thread to avoid slot advance blocking #873)
• pending-attestation replay: thread pool for XMSS verify when configured
• libp2p / network: separate runtime

Incoming gossip aggregations still run validate + XMSS verify + fork-choice store entirely on the chain-worker thread.

Proposed direction (not implemented here)

Parallel verify, serial apply — same logical model as leanSpec’s single store writer, but split CPU-bound verify from state mutation:

1. Verify stage (parallel): libxev or a small pool runs validateAttestationData + verifyAggregatedAttestation (XMSS) on cloned gossip payloads; output is a bounded queue of verified messages or an error (drop / buffer unknown-block for replay).
2. Apply stage (serial): the chain-worker (or a dedicated single “applier” thread) only runs forkChoice.onAttestation + storeAggregatedPayload for verified messages, in an order consistent with block imports (e.g. block queue still gates: no apply for targets whose state was pruned mid-import).

Do not split into three independent workers (blocks / raw att / agg) all mutating fork choice concurrently — that races STF, fork-choice head, and historical_block_hashes without a strict cross-queue ordering protocol.

Alternative considered

Two chain-worker queues (blocks vs attestations) with one writer lock on fork choice — still serial apply, only helps scheduling; does not remove XMSS cost from the hot path unless verify is moved off-thread.

Acceptance criteria

• Under devnet restart load, lean_chain_queue_dropped_total{queue="aggregated_attestation"} stays near zero without relying only on a larger queue.
• lean_pending_attestations_buffered_total{reason="queue_full"} remains bounded under sustained gossip.
• No regression in fork-choice / STF ordering (blocks still applied before dependent agg applies; replay buffer semantics unchanged).
• Document lock ordering if verify pool and chain-worker both touch states / fork choice (see docs/threading_refactor_slice_a.md, BeamNode threading model refactor — 8-point plan (supersedes #798-#802) #803).

Related

• BeamNode threading model refactor — 8-point plan (supersedes #798-#802) #803 — BeamNode threading model refactor
• Investigate slow slot_interval / tick duration (event-loop starvation vs nominal 0.8s) #863 — libxev / chain-worker move-off
• node, metrics: offload heavy chain mutations to chain-worker, parallelize XMSS verify (#863) #890 — chain-worker queues and pending replay
• node: blocks_by_range catch-up fails on fork mismatch; no retry/fallback #893 / PR node: harden blocks_by_range catch-up retry and fork recovery #894 — blocks_by_range catch-up + agg queue backpressure mitigations

Notes

Raw gossip attestations could use the same verify-then-apply split later; start with aggregations (heavier XMSS, primary queue-full source on devnet).