Skip to content

fix(ingest): stop rpc inserter spinning on the closed progress channel - #65

Open
gamandeepsingh wants to merge 1 commit into
solana-rpc:mainfrom
gamandeepsingh:fix/rpc-inserter-busy-spin-after-discovery
Open

fix(ingest): stop rpc inserter spinning on the closed progress channel#65
gamandeepsingh wants to merge 1 commit into
solana-rpc:mainfrom
gamandeepsingh:fix/rpc-inserter-busy-spin-after-discovery

Conversation

@gamandeepsingh

Copy link
Copy Markdown
Contributor

Problem

run_rpc_inserter burns 100% of a core for most of every RPC backfill.

Its select! polls progress_rx.changed() with no guard
(crates/superbank/src/ingest/rpc.rs). watch::Receiver::changed()
resolves immediately with Err once the sender is dropped, and stays
that way. Discovery drops progress_tx the moment it finishes enqueueing
slots — enqueue_slot_list / discover_slots take it by value and drop
it on return — which happens long before the workers finish fetching
those slots via getBlock. From then on the arm is permanently ready,
the select! never parks, and the loop spins.

Reproducing the loop shape standalone, counting iterations in 300ms:

iterations
progress_tx dropped (current behavior) 6,457,948
progress_tx alive (control) 2

--rpc-slot-list mode is the worst case: enqueue_slot_list only pushes
into a bounded channel, so it returns almost immediately and the spin
covers the entire fetch. That is the mode superbank-solparq --backfill-gaps runs as a subprocess. In range mode the spin starts once
discovery drains into the queue and lasts until the run ends.

Results are still inserted correctly — select! keeps choosing among
ready branches — so this shows up as CPU burn and scheduling jitter
rather than wrong data, which is likely why it went unnoticed.

Fix

Gate the arm on a progress_open flag and clear it when changed()
reports the sender is gone, so the closed channel stops being polled.
This mirrors the if !insert_tasks.is_empty() precondition already used
on the join_next arm of the same select!.

Test plan

Added inserter_parks_after_discovery_drops_progress_sender. It detects
the spin through tokio's paused clock, which auto-advances only while
every task is idle: a parked inserter lets a 30s virtual sleep resolve
instantly, a spinning one pins the clock at zero. The runtime runs on its
own thread behind a real-time deadline so a regression fails the test
instead of hanging the suite.

Confirmed in both directions:

  • with the fix — passes immediately

  • with the fix reverted — fails in 30s with
    inserter kept polling the closed progress channel instead of parking

  • cargo fmt --all -- --check

  • cargo clippy --workspace --all-targets --locked -- -D warnings

  • cargo test --workspace --locked

tokio's test-util feature is added to [dev-dependencies] for
start_paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant