Skip to content

fix(solparq): floor archived-data delete start at lowest archived slot - #70

Open
picard8472 wants to merge 2 commits into
mainfrom
fix/61-floor-delete-start-at-archived-start
Open

fix(solparq): floor archived-data delete start at lowest archived slot#70
picard8472 wants to merge 2 commits into
mainfrom
fix/61-floor-delete-start-at-archived-start

Conversation

@picard8472

Copy link
Copy Markdown
Contributor

Problem

Fixes #61.

--delete-archived-data-range always started its ClickHouse delete at slot 0. Aligned kinds don't archive from the earliest ingested slot: epoch (and custom with --custom-aligned) begin their first archive at an align_up boundary, so the window between the earliest ingested slot and that boundary is never captured by any archive.

With deletion starting at 0, that never-archived window still fell inside [0, safe_end] and got wiped — dropping data that existed in no archive. Example: ingest starts at slot 100 with epoch archiving; the first epoch archive is [432000, 863999], but the delete swept [0, 863999], so 100..431999 was gone from ClickHouse and in no archive. (Original review comment.)

Fix

Floor the delete start at the lowest archived start slot across the configured kinds instead of 0. Any slot at or above it exists in at least one archive and can be safely reclaimed; the never-archived leading window stays in ClickHouse. safe_end (the min of each kind's latest archive end) is unchanged, so the "delete only once every kind has covered it" high-watermark still holds.

For setups that intentionally want the leading, never-archived window purged too, add --delete-archived-data-from-slot-zero (env SOLPARQ_DELETE_ARCHIVED_DATA_FROM_SLOT_ZERO) — server mode only, off by default — which restores sweeping from slot 0.

Changes

  • safe_delete_archived_data_range now tracks min(parsed.start_slot) alongside min(parsed.end_slot) and floors the delete start there (clamped to safe_end), unless the new opt-out flag is set.
  • New --delete-archived-data-from-slot-zero CLI flag + Config field, gated to --server-mode (errors otherwise). Exposed in the ops JSON.
  • Updated crates/superbank-solparq/README.md (Cleanup section, aligned-custom caveat, Options list).

Testing

  • cargo test -p superbank-solparq --locked — all pass. Updated the epoch-continuation cleanup test to expect the floored start, and added tests for the from-slot-zero opt-in and the server-mode-only validation.
  • cargo clippy -p superbank-solparq --all-targets --locked -- -D warnings and cargo fmt --all -- --check — clean.

🤖 Generated with Claude Code

picard8472 and others added 2 commits August 5, 2026 14:06
Deletion of archived ClickHouse data previously always started at slot 0.
Aligned kinds (epoch, or custom with --custom-aligned) begin their first
archive at an align_up boundary, so the window between the earliest
ingested slot and that boundary is never captured by any archive. With
--delete-archived-data-range on, that never-archived window still fell
inside [0, safe_end] and was deleted, dropping data that existed in no
archive.

Floor the delete start at the lowest archived start slot across the
configured kinds instead of 0. Any slot at or above it exists in at least
one archive and can be safely reclaimed.

Add --delete-archived-data-from-slot-zero (server mode only, off by
default) to opt back into sweeping from slot 0 for setups that want the
leading, never-archived window purged too.

Fixes #61.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flag "--delete-archived-data-range" can delete slots that were never archived

1 participant