Progressive data structures and tests EIP-7916, EIP-8016

[michaelsproul]

Proposed Changes

Run the EF tests for new types from EIP-7916 and EIP-8016:

• ProgressiveList
• ProgressiveBitList
• Progressive containers
• Compatible unions

The SSZ generic tests are passing if you run them without compiling the rest of Lighthouse:

cargo test -p ef_tests --release --features ef_tests -- ssz_generic

Everything else is probably broken.

Additional Info

Depends on:

• Progressive list milhouse#82
• Progressive bitlist and compatible union support EIP-7916, EIP-8016 ethereum_ssz#67
• ProgressiveMerkleHasher and progressive container support tree_hash#43

Known Issues

• The introduction of BitList<Progressive>::with_capacity has created some type inference ambiguities causing state_processing not to compile. We should probably rename this method in ethereum_ssz.
• Haven't merged unstable since the types refactor nor the gloas-containers merge.

[michaelsproul]

I've made a bunch of progress on this, but it requires local checkouts of ethereum_ssz and tree_hash at the moment. These branches:

• Progressive bitlist and compatible union support EIP-7916, EIP-8016 ethereum_ssz#67
• ProgressiveMerkleHasher and progressive container support tree_hash#43

Progressive bitlist tests are currently not passing, need to dig into why. The derive macro for progressive containers with 1 field works though 😅

[michaelsproul]

Looking for feedback on the UX of this.

I quite like this UX for progressive containers because it closely follows the spec. We can just copy the active_fields vec straight into the struct attributes.

[Zyra-V21]

Hi @michaelsproul (cc @macladson) !! I've been mapping SSZ progressive-type support across the Rust ecosystem for downstream EL adoption. alloy is currently missing ProgressiveList<Withdrawal>, which blocks EIP-6465 in Reth, and the same dependency holds for EIP-6404 / EIP-6466. The stack here (ethereum_ssz#67, tree_hash#43, milhouse#82, this PR) covers everything needed, and per your description the ssz_generic EF tests are already passing.

Reading the four PRs together, the visible blockers seem to be:

1. Mac's note on tree_hash#43 about flipping the tree structure to the new spec , has the spec rev landed, and is the rework still pending?
2. The BitList<Progressive>::with_capacity type-inference issue you flagged here, plus the unstable rebase debt across the four PRs.
3. ~24% patch-coverage gap on ethereum_ssz#67 (mostly bitfield/progressive.rs), and the compile-fail tests you were considering doc-test based.

If priorities pulled you elsewhere and you'd welcome help, I'd be glad to take on the rebase + spec-sync across the four PRs and close out those blockers. I'd do the work as draft PRs from a fork so you keep ownership of the design , happy to squash credit back to you and Mac on merge. Won't touch anything until you say go.

Two questions before I'd start:

1. Is the spec change Mac referenced documented anywhere I can read (consensus-specs commit, EIPs PR, or design discussion)?
2. Are there sigp-internal priorities (gloas rollout, Amsterdam timing) that should shape the order I tackle the four PRs?

Happy to move this to Discord if easier.

[michaelsproul]

Hi @Zyra-V21 we've got it covered

[michaelsproul]

Closing in favour of:

• EIP-7688: Forward Compatible Consensus Data Structures #9450