Skip to content

Add Distributed Blob Publishing to Fulu networking spec #4183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Mar 28, 2025
Merged

Add Distributed Blob Publishing to Fulu networking spec #4183

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
ebec514
Add distributed blob publishing section to Fulu p2p interface spec.
jimmygchen Mar 21, 2025
2739e5f
Apply suggestions from code review
jimmygchen Mar 21, 2025
a027ff6
Apply suggestions from code review
jimmygchen Mar 24, 2025
a74fa60
Fix lint
jimmygchen Mar 24, 2025
78bde87
Fix lint.
jimmygchen Mar 24, 2025
6ec7d3f
Merge branch 'dev' into distributed-blob-publishing
jimmygchen Mar 24, 2025
f798950
Add new line before list
jtraglia Mar 25, 2025
43892f0
Clarify columns that clients must publish to in Distributed Publishing.
jimmygchen Mar 26, 2025
dc25211
add function to retrieve sidecars from both block and sidecar
fradamt Mar 26, 2025
34706cf
fix lint
fradamt Mar 26, 2025
3a232f6
fix lint again
fradamt Mar 26, 2025
f208942
fix tests
fradamt Mar 26, 2025
b81b6eb
Fix tests & nits
jtraglia Mar 26, 2025
b57e22a
Fix more nits
jtraglia Mar 26, 2025
053f03b
Merge branch 'dev' into distributed-blob-publishing
jimmygchen Mar 27, 2025
f32d33d
Address review comment.
jimmygchen Mar 27, 2025
c76082d
clarify language
fradamt Mar 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 9 additions & 10 deletions specs/deneb/p2p-interface.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,15 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`:
|--------------------------------|---------------------|
| `DENEB_FORK_VERSION` and later | `deneb.BlobSidecar` |

###### Blob retrieval via local execution layer client

In addition to `BlobSidecarsByRoot` requests, recent blobs MAY be retrieved by querying the Execution Layer (i.e. via `engine_getBlobsV1`).
Implementers are encouraged to leverage this method to increase the likelihood of incorporating and attesting to the last block when its proposer is not able to publish blobs on time.

When clients use the local execution layer to retrieve blobs, they MUST behave as if the corresponding `blob_sidecar` had been received via gossip. In particular they MUST:
* publish the corresponding `blob_sidecar` on the `blob_sidecar_{subnet_id}` subnet.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need to publish blob_sidecar after Fulu? My understanding is that after it'll only be data column sidecars?

Copy link
Contributor Author

@jimmygchen jimmygchen Mar 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No we don't need to publsih blob_sidecars after Fulu. This is the deneb specification - I've only moved this section (from RPC to gossip) without modification.

* update gossip rule related data structures (i.e. update the anti-equivocation cache).

##### Attestation subnets

###### `beacon_attestation_{subnet_id}`
Expand Down Expand Up @@ -413,16 +422,6 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`:
|--------------------------------|---------------------|
| `DENEB_FORK_VERSION` and later | `deneb.BlobSidecar` |

###### Blob retrieval via local execution layer client

In addition to `BlobSidecarsByRoot` requests, recent blobs MAY be retrieved by querying the Execution Layer (i.e. via `engine_getBlobsV1`).
Implementers are encouraged to leverage this method to increase the likelihood of incorporating and attesting to the last block when its proposer is not able to publish blobs on time.

When clients use the local execution layer to retrieve blobs, they MUST behave as if the corresponding `blob_sidecar` had been received via gossip. In particular they MUST:

* publish the corresponding `blob_sidecar` on the `blob_sidecar_{subnet_id}` subnet.
* update gossip rule related data structures (i.e. update the anti-equivocation cache).

## Design decision rationale

### Why are blobs relayed as a sidecar, separate from beacon blocks?
Expand Down
11 changes: 11 additions & 0 deletions specs/fulu/p2p-interface.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
- [Blob subnets](#blob-subnets)
- [Deprecated `blob_sidecar_{subnet_id}`](#deprecated-blob_sidecar_subnet_id)
- [`data_column_sidecar_{subnet_id}`](#data_column_sidecar_subnet_id)
- [Distributed Blob Publishing using blobs retreived from local execution layer client](#distributed-blob-publishing-using-blobs-retreived-from-local-execution-layer-client)
- [The Req/Resp domain](#the-reqresp-domain)
- [Messages](#messages)
- [DataColumnSidecarsByRange v1](#datacolumnsidecarsbyrange-v1)
Expand Down Expand Up @@ -204,6 +205,16 @@ The following validations MUST pass before forwarding the `sidecar: DataColumnSi

*Note:* In the `verify_data_column_sidecar_inclusion_proof(sidecar)` check, for all the sidecars of the same block, it verifies against the same set of `kzg_commitments` of the given beacon block. Client can choose to cache the result of the arguments tuple `(sidecar.kzg_commitments, sidecar.kzg_commitments_inclusion_proof, sidecar.signed_block_header)`.

###### Distributed Blob Publishing using blobs retreived from local execution layer client

Similar to the Deneb specification, recent blobs MAY be retrieved by querying the Execution Layer via `engine_getBlobsV2`. Clients will be able to convert the EL blob response into data columns to satisfy block availability requirements.

Implementers are encouraged to leverage this method to increase the likelihood of incorporating and attesting to the last block when its proposer is not able to publish data columns on time.

When clients use the local execution layer to retrieve blob and compute data columns, they MUST behave as if the imported `data_column_sidecar` had been received via gossip. Note that this only applies to the data columns gossip topics they are subscribed to - specifically, the node’s assigned custody columns. In particular, clients MUST:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jimmygchen i would like clarify on this statement. when full node participate in distributed blob publishing, does the node publish only its custody columns, or plus some extra sampling columns? since full node also subscribe those extra sampling columns subnets, not only custody subnets

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @hangleang, I can try to help with this. The full node would only publish the data column sidecars to the subnets for which it is custodying, which is determined by get_custody_groups .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jtraglia - I realise what I wrote there isn't right for full nodes without validators, the following sentence should be removed / replaced:

specifically, the node’s assigned custody columns.

Because under subnet sampling, a full node without validators would subscribe to 8 column topics but only custody 4 columns - and distributed publishing should apply to all subscribed column topics to make sure columns are propagated accordingly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see. Then I misunderstood it too 😅 good question @hangleang!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 43892f0, hope this makes sense:

consensus-specs/specs/fulu/p2p-interface.md

Lines 214 to 217 in 43892f0

When clients use the local execution layer to retrieve blob and compute data columns, they MUST behave as if the imported `data_column_sidecar` had been received via gossip. In particular, clients MUST:
* Publish the corresponding `data_column_sidecar` on the `data_column_sidecar_{subnet_id}` topic **if and only if** they are **subscribed** to it, either due to custody requirement or additional sampling.
* Update gossip rule related data structures (i.e. update the anti-equivocation cache).

* publish the corresponding `data_column_sidecar` on the `data_column_sidecar_{subnet_id}` subnet.
* update gossip rule related data structures (i.e. update the anti-equivocation cache).

### The Req/Resp domain

#### Messages
Expand Down