fix(solparq): floor archived-data delete start at lowest archived slot - #70
Open
picard8472 wants to merge 2 commits into
Open
fix(solparq): floor archived-data delete start at lowest archived slot#70picard8472 wants to merge 2 commits into
picard8472 wants to merge 2 commits into
Conversation
Deletion of archived ClickHouse data previously always started at slot 0. Aligned kinds (epoch, or custom with --custom-aligned) begin their first archive at an align_up boundary, so the window between the earliest ingested slot and that boundary is never captured by any archive. With --delete-archived-data-range on, that never-archived window still fell inside [0, safe_end] and was deleted, dropping data that existed in no archive. Floor the delete start at the lowest archived start slot across the configured kinds instead of 0. Any slot at or above it exists in at least one archive and can be safely reclaimed. Add --delete-archived-data-from-slot-zero (server mode only, off by default) to opt back into sweeping from slot 0 for setups that want the leading, never-archived window purged too. Fixes #61. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #61.
--delete-archived-data-rangealways started its ClickHouse delete at slot0. Aligned kinds don't archive from the earliest ingested slot:epoch(andcustomwith--custom-aligned) begin their first archive at analign_upboundary, so the window between the earliest ingested slot and that boundary is never captured by any archive.With deletion starting at
0, that never-archived window still fell inside[0, safe_end]and got wiped — dropping data that existed in no archive. Example: ingest starts at slot100with epoch archiving; the first epoch archive is[432000, 863999], but the delete swept[0, 863999], so100..431999was gone from ClickHouse and in no archive. (Original review comment.)Fix
Floor the delete start at the lowest archived start slot across the configured kinds instead of
0. Any slot at or above it exists in at least one archive and can be safely reclaimed; the never-archived leading window stays in ClickHouse.safe_end(the min of each kind's latest archive end) is unchanged, so the "delete only once every kind has covered it" high-watermark still holds.For setups that intentionally want the leading, never-archived window purged too, add
--delete-archived-data-from-slot-zero(envSOLPARQ_DELETE_ARCHIVED_DATA_FROM_SLOT_ZERO) — server mode only, off by default — which restores sweeping from slot0.Changes
safe_delete_archived_data_rangenow tracksmin(parsed.start_slot)alongsidemin(parsed.end_slot)and floors the delete start there (clamped tosafe_end), unless the new opt-out flag is set.--delete-archived-data-from-slot-zeroCLI flag +Configfield, gated to--server-mode(errors otherwise). Exposed in the ops JSON.crates/superbank-solparq/README.md(Cleanup section, aligned-custom caveat, Options list).Testing
cargo test -p superbank-solparq --locked— all pass. Updated the epoch-continuation cleanup test to expect the floored start, and added tests for the from-slot-zero opt-in and the server-mode-only validation.cargo clippy -p superbank-solparq --all-targets --locked -- -D warningsandcargo fmt --all -- --check— clean.🤖 Generated with Claude Code