Skip to content

Add archiveSidecarsProofs to Database: persist KZG proofs and drop extension columns - #11250

Open
zilm13 wants to merge 5 commits into
Consensys-Incorporated:masterfrom
zilm13:half-storage-database
Open

zilm13 wants to merge 5 commits into
Consensys-Incorporated:masterfrom
zilm13:half-storage-database

Conversation

@zilm13

@zilm13 zilm13 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

PR Description

Another part of #10132
No activation yet, just methods that will be called later + tests.

Fixed Issue(s)

Documentation

  • I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

  • I thought about adding a changelog entry, and added one if I deemed necessary.

Note

Medium Risk
Changes persisted blob/column storage and pruning behavior for Fulu data; incorrect completeness checks or prune/proof cleanup could make extension columns unrecoverable or leave stale proof entries.

Overview
Adds archiveSidecarsProofs on Database / KvStoreDatabase to shrink Fulu storage: for each slot in a range with a complete first and extension half of data columns, it stores extension-column KZG proofs, deletes extension sidecars, and leaves the first-half columns for later reconstruction. Slots with any missing column in either half are skipped so archived data stays recoverable.

Canonical pruneAllSidecars now also removes archived proofs for pruned slots (with debug logging), keeping proof metadata aligned with sidecar pruning.

Includes integration coverage in DatabaseTest, serializer edge-case tests, and CombinedChainDataClient tests for reading stored proofs. NoOpDatabase stub added; wiring through the pruner is left for follow-up (not activated in this PR).

Reviewed by Cursor Bugbot for commit c2f5360. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit fc73777. Configure here.

// Skip slots where either half is incomplete: first-half columns are required for
// reconstruction, and archiving extension columns without them would make the archived
// data irrecoverable.
if (extensionKeys.size() != halfColumns || firstHalfCount != halfColumns) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Archive completeness ignores block root

Medium Severity

archiveSidecarsProofs groups identifiers and checks half-completeness by slot only, while sidecars are keyed by (slot, blockRoot, columnIndex) and proofs are stored per slot. Canonical storage can hold columns from more than one root at the same slot (onNewSidecar does not evict others). Extra leftovers make both halves look over-complete and the slot is skipped, so a fully populated canonical set is never archived. Complementary halves from two roots can still pass the halfColumns checks, persist mixed proofs, and delete extension columns that reconstruction cannot recover.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit fc73777. Configure here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the feature works only for finalized sidecars
it's impossible to finalize several roots for one slot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant