Skip to content

fix: confirmed slot gap during startup causes unrecoverable crash-loop - #27

Open
gamandeepsingh wants to merge 1 commit into
solana-rpc:mainfrom
gamandeepsingh:fix/startup-gap-crash-loop
Open

fix: confirmed slot gap during startup causes unrecoverable crash-loop#27
gamandeepsingh wants to merge 1 commit into
solana-rpc:mainfrom
gamandeepsingh:fix/startup-gap-crash-loop

Conversation

@gamandeepsingh

Copy link
Copy Markdown

fix: confirmed slot gap during startup causes unrecoverable crash-loop

On large databases the post-snapshot pipeline (dedup, index creation, cleanup) runs for several hours with finalization paused. During that window the gRPC buffer fills up, back-pressures the stream, and the upstream drops a slot — triggering a confirmed gap. check_slot_gap then paused the finalizer, which is the very worker that drives startup to completion, creating a deadlock. fill_gaps detected startup wasn't done and panic!'d, restarting the entire process and re-downloading the full snapshot. On a ~108 GB mainnet index this loop is unrecoverable.

Fix:

  • SelfHealingState now holds a reference to snapshot_processing_state. In check_slot_gap, the finalizer is only paused if startup is already finished — gaps detected during startup are still queued for repair but the finalizer keeps running.
  • In fill_gaps, the panic! is replaced with continue. The loop retries every 30s until startup finishes, then pauses the finalizer and repairs the queued gaps through the normal path.

Closes: #19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Confirmed slot gap during startup panics the indexer, causing an unrecoverable re-bootstrap loop on large databases

1 participant