fix(ingest): stop rpc inserter spinning on the closed progress channel - #65
Open
gamandeepsingh wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
run_rpc_inserterburns 100% of a core for most of every RPC backfill.Its
select!pollsprogress_rx.changed()with no guard(
crates/superbank/src/ingest/rpc.rs).watch::Receiver::changed()resolves immediately with
Erronce the sender is dropped, and staysthat way. Discovery drops
progress_txthe moment it finishes enqueueingslots —
enqueue_slot_list/discover_slotstake it by value and dropit on return — which happens long before the workers finish fetching
those slots via
getBlock. From then on the arm is permanently ready,the
select!never parks, and the loop spins.Reproducing the loop shape standalone, counting iterations in 300ms:
progress_txdropped (current behavior)progress_txalive (control)--rpc-slot-listmode is the worst case:enqueue_slot_listonly pushesinto a bounded channel, so it returns almost immediately and the spin
covers the entire fetch. That is the mode
superbank-solparq --backfill-gapsruns as a subprocess. In range mode the spin starts oncediscovery drains into the queue and lasts until the run ends.
Results are still inserted correctly —
select!keeps choosing amongready branches — so this shows up as CPU burn and scheduling jitter
rather than wrong data, which is likely why it went unnoticed.
Fix
Gate the arm on a
progress_openflag and clear it whenchanged()reports the sender is gone, so the closed channel stops being polled.
This mirrors the
if !insert_tasks.is_empty()precondition already usedon the
join_nextarm of the sameselect!.Test plan
Added
inserter_parks_after_discovery_drops_progress_sender. It detectsthe spin through tokio's paused clock, which auto-advances only while
every task is idle: a parked inserter lets a 30s virtual sleep resolve
instantly, a spinning one pins the clock at zero. The runtime runs on its
own thread behind a real-time deadline so a regression fails the test
instead of hanging the suite.
Confirmed in both directions:
with the fix — passes immediately
with the fix reverted — fails in 30s with
inserter kept polling the closed progress channel instead of parkingcargo fmt --all -- --checkcargo clippy --workspace --all-targets --locked -- -D warningscargo test --workspace --lockedtokio'stest-utilfeature is added to[dev-dependencies]forstart_paused.