Skip to content

fix(storage): compute tombstone log resume slot from global offset - #21

Open
detail-app[bot] wants to merge 2 commits into
mainfrom
detail/bug-fix/fix-storage-compute-tombstone-log-resume-slot-from-b5a8d8
Open

detail-app[bot] wants to merge 2 commits into
mainfrom
detail/bug-fix/fix-storage-compute-tombstone-log-resume-slot-from-b5a8d8

Conversation

@detail-app

@detail-app detail-app Bot commented Sep 11, 2026

Copy link
Copy Markdown

Detail bug report: View on Detail

Bug

TombstoneLog::open computed the writer's resume slot from a per-page-local byte offset. Inside the recovery loop, addr was assigned slot * SERIALIZED_LEN — always < PAGE — so latest_tombstone_page = addr / PAGE was always 0 and the cross-page else branch that converts the offset to a global slot was unreachable dead code.

After a reopen where the most-recent tombstone lives on a non-zero page (a multi-page tombstone log with more than SLOTS_PER_PAGE tombstones appended since the last open), the writer resumed on page 0 instead of right after the last global slot. The next append then overwrote an older, still-needed tombstone on page 0. If that overwritten tombstone was the delete marker for a key whose data block was still on disk (deletes are logical; blocks persist until reclaimed), the next reopen re-indexed the stale block and load returned a deleted value — a data-integrity violation (phantom entry).

This regressed in the partition-abstraction refactor, which dropped the page-base addend from addr. The bug only affects deployments with enable_tombstone_log: true (off by default).

Fix

foyer-storage/src/engine/block/tombstone.rs: track a global byte offset across all partitions and pages. A global_addr accumulator advances by SERIALIZED_LEN for every slot (empty or not); addr (offset of the running most-recent tombstone) is set to global_addr instead of the page-local slot * SERIALIZED_LEN. seq/addr are hoisted out of the page loop so they track the global most-recent tombstone. The formerly-dead else branch now yields the correct global slot, and slot = that + 1 resumes immediately after the most-recent tombstone on both first-lap and wrapped-lap positions.

Testing

  • Added test_tombstone_log_resume_after_wrap (tombstone.rs): writes 1290 tombstones into a 4-page (1024-slot) log so the ring wraps once, reopens, and asserts the resume slot is the global 267 (mapped to physical page 1) rather than the page-local 11 (page 0).
  • Added test_store_tombstone_log_resume_no_phantom_entry (engine.rs): a 5-page tombstone-log end-to-end test that inserts+flushes an entry, deletes it, pushes the most-recent tombstone onto page 1, reopens, appends one more tombstone, reopens again, and asserts load stays None (no phantom deleted value). Uses a new store_for_test_with_multipage_tombstone_log helper.
  • Both new tests were verified to fail under the reintroduced bug (resume 11 vs 267; phantom Some((1, ...)) vs None) and pass under the fix, confirming they genuinely depend on the fix.
  • No regressions: all existing engine::block tests pass, including test_store_delete_recovery, test_store_destroy_recovery, and test_tombstone_log (single-page and page-0-most-recent configs where the bug coincidentally didn't manifest).
  • Routine checks pass: cargo check -p foyer-storage, cargo fmt -- --check on both files, and cargo clippy -p foyer-storage introduce no new warnings (a pre-existing load_throttle_switch dead-code warning in store.rs is unrelated to this change).
  • Full foyer-storage test suite with --features test_utils passes (26/26, including the three integration fuzzy tests).
  • The --features serde clippy matrix fails, but this is a pre-existing failure (foyer_common::properties::Age missing Serialize/Deserialize impls) unrelated to this fix and reproducible on the clean tree.

Automatic Fixes PRs can be configured here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants