Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EIP7594: p2p-interface #6358

Draft
wants to merge 110 commits into
base: kzgpeerdas
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
02e5430
init: add req/res domain for peerdas
agnxsh Jun 14, 2024
986a2bd
save work push, build failing
agnxsh Jun 14, 2024
5934400
add: req/res rpc handlers
agnxsh Jun 14, 2024
ebe9b3b
rm: TODO comment, revisiting later
agnxsh Jun 15, 2024
6064ac3
add tests for sanity checking data columns
agnxsh Jun 17, 2024
8e49f88
update URLs
agnxsh Jun 17, 2024
46d07b1
add: data column support in sync_protocol, sync_manager, request_mana…
agnxsh Jun 18, 2024
c0eb4c4
update test report for db test
agnxsh Jun 18, 2024
51f189e
add: getMissingDataColumns, requestManagerDataColumnLoop
agnxsh Jun 18, 2024
b87a6d7
Merge branch 'peerdas-p2p' of https://github.com/status-im/nimbus-eth…
agnxsh Jun 18, 2024
f0cae30
add: pruneDataColumns at the end of slot
agnxsh Jun 19, 2024
e2afc58
fix: reviews, pass1
agnxsh Jun 21, 2024
791d2fb
add: forward and backward syncing for data columns, broadcasting data…
agnxsh Jun 24, 2024
9bdcd5e
fix: sync tests
agnxsh Jun 24, 2024
07d33b3
add dataColumns to db during forward syncing
agnxsh Jun 24, 2024
325bdfd
support for enqueueing whichever is activated blob/data_column
agnxsh Jun 25, 2024
aa390e9
rm: message router logic for data column, need to move it
agnxsh Jun 25, 2024
81b55fa
add: fetch subnetCount for super node when subscribeAllSubnets flag p…
agnxsh Jun 25, 2024
87bc91f
fix: message router
agnxsh Jun 25, 2024
18e3ba2
fix: get_data_column_sidecar
agnxsh Jun 26, 2024
0b4cf10
rm: unused code in data column getter
agnxsh Jun 26, 2024
34a2478
add: blob recovery logic
agnxsh Jun 26, 2024
3db92f8
add: data column reconstruction logic
agnxsh Jun 27, 2024
5bf1e02
initiate data column quarantine
agnxsh Jun 28, 2024
27b0705
verify kzg disable
agnxsh Jun 28, 2024
0e01d2f
experimental disable for inclusion proofs
agnxsh Jun 28, 2024
ca3bd3e
experimental: disable scoring for data columns
agnxsh Jun 28, 2024
7426690
dc quarantine activation, keeping blobs compatible
agnxsh Jun 29, 2024
c8d957a
add: experimental checkpoints on gossip validation to localize failin…
agnxsh Jun 29, 2024
8292341
fix: block_processor test
agnxsh Jun 29, 2024
510d988
disable subnet gossip condition, fixed inclusion proof
agnxsh Jun 30, 2024
26ac587
request man for data columns
agnxsh Jul 1, 2024
8ac4cc9
add: data column grouping conditions for range request
agnxsh Jul 1, 2024
8e28654
exp: build failing, checking if failing on other machines with these …
agnxsh Jul 1, 2024
3b1f5b4
weird fix
agnxsh Jul 1, 2024
9325423
strangely disable this line makes it go away :)
agnxsh Jul 1, 2024
0e02eb4
fix test_sync_manager
agnxsh Jul 1, 2024
a8e2c3e
exp: disable some gossip conditions
agnxsh Jul 1, 2024
9e6cad4
bit more disabling for kurtosis
agnxsh Jul 1, 2024
67fe8ac
disable blob activity (exp), improve gossip validation
agnxsh Jul 2, 2024
2f7a3d0
reenable checkpoints to debug exception
agnxsh Jul 2, 2024
75c3e0b
debug
agnxsh Jul 2, 2024
77cc2ef
debug2
agnxsh Jul 2, 2024
8d2c489
debug3
agnxsh Jul 2, 2024
d8e1bef
debug 4
agnxsh Jul 2, 2024
d0722cd
update constants
agnxsh Jul 2, 2024
0e710da
update timings
agnxsh Jul 2, 2024
8f9f654
intentionally increase custody requirement
agnxsh Jul 2, 2024
ad64b22
shortLog for dc
agnxsh Jul 2, 2024
d292e94
sync queue
agnxsh Jul 2, 2024
152d276
added reconstruction logic
agnxsh Jul 3, 2024
9f42196
exp disable of some gossip conditions
agnxsh Jul 3, 2024
53f7175
minor fix
agnxsh Jul 3, 2024
887a44a
revert gossip val
agnxsh Jul 3, 2024
7063739
fix: get_data_column_sidecars
agnxsh Jul 3, 2024
d49b1a1
fix: ckzg function change
agnxsh Jul 3, 2024
1a85760
fix: cell and proof aggregator
agnxsh Jul 3, 2024
c6662bd
reenable blobs in block proposal
agnxsh Jul 4, 2024
a755dba
inclusion proof depth
agnxsh Jul 4, 2024
3bea574
reduce data column response cost
agnxsh Jul 4, 2024
b927ddd
fix: get_data_column_sidecars
agnxsh Jul 4, 2024
13029d9
fix: get data column
agnxsh Jul 4, 2024
41b35b9
fix: get data column fixes
agnxsh Jul 4, 2024
fe183e7
change timings
agnxsh Jul 4, 2024
24b30a9
test kurtosis
agnxsh Jul 4, 2024
93c3525
increase ops cost
agnxsh Jul 4, 2024
6cdc6bf
debug: verify data column kzg proofs via kurtosis
agnxsh Jul 4, 2024
eb46f4c
fix kzg inclusion proof logic
agnxsh Jul 4, 2024
14afc82
gindex fix
agnxsh Jul 4, 2024
318d656
fix: gindex
agnxsh Jul 5, 2024
fab427d
enable dc in gossip and message router
agnxsh Jul 5, 2024
086d3f1
fix: get dc sidecar
agnxsh Jul 5, 2024
778ea9f
fix build proof in get dc
agnxsh Jul 5, 2024
a92eda5
prevent pulling const values from deneb preset
agnxsh Jul 5, 2024
7101f93
gindex issue fix
agnxsh Jul 5, 2024
adc717c
change return type for get dc
agnxsh Jul 5, 2024
1729bdc
reduce parallel requests
agnxsh Jul 5, 2024
85db9ca
regressive fix
agnxsh Jul 5, 2024
74ee8bb
refactor cells and proofs logic + fix edge cases
agnxsh Jul 6, 2024
7b9c68b
oops
agnxsh Jul 6, 2024
755c24d
fix: blob len 0 case
agnxsh Jul 6, 2024
abf5892
handle empty blobs
agnxsh Jul 6, 2024
e5237d1
cell and proof extraction
agnxsh Jul 6, 2024
c14b592
add: checkpoints for debug support
agnxsh Jul 6, 2024
7a891f1
rework on cell and proof
agnxsh Jul 6, 2024
0bffdd0
change checkpoints
agnxsh Jul 6, 2024
aaba448
convert to List add
agnxsh Jul 7, 2024
5eb854b
checkpoint 2 cleanup
agnxsh Jul 7, 2024
cf40d7f
cleanup for debugs, complete
agnxsh Jul 8, 2024
b33900b
added enr struct
agnxsh Jul 15, 2024
e034f30
add: subscribeAllSubnets feature
agnxsh Jul 15, 2024
fa5b154
add: logic constructing valid set of peers
agnxsh Jul 19, 2024
5265eeb
refactor: sync manager to range request only valid peers if not super…
agnxsh Jul 22, 2024
d2c7208
Eth2Node not needed in sync man
agnxsh Jul 22, 2024
b001499
add: valid custody peer set to RequestManager
agnxsh Jul 23, 2024
c651312
fix reviewed code
agnxsh Jul 23, 2024
7faec9b
nits
agnxsh Jul 23, 2024
8744888
add: hypergeom cdf
agnxsh Jul 25, 2024
2e9750b
add: get_extended_sample_count for lossy sampler and it's unit test
agnxsh Jul 25, 2024
e80bd36
add: verify data column kzg proof during storeBlock, added serializeD…
agnxsh Jul 26, 2024
329fc21
add: condition for being able to selfReconstruct
agnxsh Jul 29, 2024
20e6b18
resovle merge conflicts
agnxsh Jul 29, 2024
b32205d
upstream peerdas alpha3 related spec changes + fix upstream related i…
agnxsh Aug 5, 2024
9be615d
add: data column reconstruction and broadcast (#6481)
agnxsh Aug 8, 2024
1ebba1f
add: metadata-v3 for custody subnet count (#6486)
agnxsh Aug 12, 2024
249eb0e
bump nim-bearssl to 646fa2152b11980c24bf34b3e214b479c9d25f21
agnxsh Aug 13, 2024
722480a
bumped nim-chronos to 1b9d9253e89445d585d0fff39cc0d19254fdfd0d
agnxsh Aug 13, 2024
65a5255
change gcc config to tackle incompatible pointer types
agnxsh Aug 13, 2024
cc21a2a
fix: enr bitfield logic for custody subnet count
agnxsh Aug 14, 2024
f3f61cb
conditionally reconstruct and broadcast only when supernode
agnxsh Aug 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 100 additions & 3 deletions beacon_chain/networking/eth2_network.nim
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ import
eth/net/nat, eth/p2p/discoveryv5/[enr, node, random2],
".."/[version, conf, beacon_clock, conf_light_client],
../spec/datatypes/[phase0, altair, bellatrix, eip7594],
../spec/[eth2_ssz_serialization, network, helpers, forks],
../spec/[eth2_ssz_serialization, network,
helpers, eip7594_helpers, forks],
../validators/keystore_management,
"."/[eth2_discovery, eth2_protocol_dsl, libp2p_json_serialization, peer_pool, peer_scores]

Expand Down Expand Up @@ -1482,7 +1483,8 @@ proc trimConnections(node: Eth2Node, count: int) =
inc(nbc_cycling_kicked_peers)
if toKick <= 0: return

proc getLowSubnets(node: Eth2Node, epoch: Epoch): (AttnetBits, SyncnetBits, CscBits) =
proc getLowSubnets(node: Eth2Node, epoch: Epoch):
(AttnetBits, SyncnetBits, CscBits) =
# Returns the subnets required to have a healthy mesh
# The subnets are computed, to, in order:
# - Have 0 subnet with < `dLow` peers from topic subscription
Expand Down Expand Up @@ -1575,7 +1577,11 @@ proc runDiscoveryLoop(node: Eth2Node) {.async.} =
else:
0
discoveredNodes = await node.discovery.queryRandom(
node.discoveryForkId, wantedAttnets, wantedSyncnets, wantedCscnets, minScore)
node.discoveryForkId,
wantedAttnets,
wantedSyncnets,
wantedCscnets,
minScore)

let newPeers = block:
var np = newSeq[PeerAddr]()
Expand Down Expand Up @@ -1624,6 +1630,97 @@ proc runDiscoveryLoop(node: Eth2Node) {.async.} =
# Also, give some time to dial the discovered nodes and update stats etc
await sleepAsync(5.seconds)

proc getNodeIdFromPeer*(peer: Peer): NodeId=
# Convert peer id to node id by extracting the peer's public key
let nodeId =
block:
var key: PublicKey
discard peer.peerId.extractPublicKey(key)
keys.PublicKey.fromRaw(key.skkey.getBytes()).get().toNodeId()
nodeId

proc fetchCustodyColumnCountFromRemotePeer*(node: Eth2Node,
pid: PeerId):
uint64 =
# Fetches the custody column count from a remote peer
# if the peer advertises their custody column count
# via the `csc` ENR field. If the peer does NOT, then
# the default value is assume, i.e, CUSTODY_REQUIREMENT
let
eth2node = node
peer = eth2node.getPeer(pid)

let enrOpt = peer.enr
if enrOpt.isNone:
debug "Could not get ENR from peer",
peer_id = pid
return 0

else:
let
enr = enrOpt.get
enrFieldOpt =
enr.get(enrCustodySubnetCountField, uint64)

if not enrFieldOpt.isOk:
debug "Issue with fetching `csc` field from ENR",
enr = enr
else:
return(enrFieldOpt.get)

proc constructValidCustodyPeers*(node: Eth2Node,
peers: openArray[Peer],
config: BeaconNodeConf):
seq[PeerId] =
# `constructValidaCustodyPeers` returns a list of peers that are an overall
# superset of the given local node's custody columns and the custody columns
# of the remote peers

# Get the local custody subnet count
var localCustodySubnetCount: uint64
if config.subscribeAllSubnets:
localCustodySubnetCount = DATA_COLUMN_SIDECAR_SUBNET_COUNT.uint64
localCustodySubnetCount = CUSTODY_REQUIREMENT

# Get the local custody columns
let
localNodeId = node.nodeId
localCustodyColumns =
localNodeId.get_custody_columns(localCustodySubnetCount).get

var validPeerIds = newSeqOfCap[PeerId](peers.len)

for peer in peers:
# Get the custody subnet counts of the remote peer
let remoteCustodySubnetCount =
node.fetchCustodyColumnCountFromRemotePeer(peer.peerId)

# Extract remote peer's nodeID from peerID
# Fetch custody columns fromm remote peer
let
remoteNodeId = getNodeIdFromPeer(peer)
remoteCustodyColumns =
remoteNodeId.get_custody_columns(remoteCustodySubnetCount).get

# If the remote peer custodies less columns than our local node
# We skip it
if remoteCustodyColumns.len < localCustodyColumns.len:
continue

# If the remote peer custodies all the possible columns
if remoteCustodyColumns.len == NUMBER_OF_COLUMNS:
validPeerIds.add(peer.peerId)

# Filtering out the invalid peers
for column in localCustodyColumns:
if column notin remoteCustodyColumns:
continue

# Otherwise add the peer to the set
# of valid peerIds
validPeerIds.add(peer.peerId)
validPeerIds

proc resolvePeer(peer: Peer) =
# Resolve task which performs searching of peer's public key and recovery of
# ENR using discovery5. We only resolve ENR for peers we know about to avoid
Expand Down
26 changes: 0 additions & 26 deletions beacon_chain/nimbus_beacon_node.nim
Original file line number Diff line number Diff line change
Expand Up @@ -1136,32 +1136,6 @@ proc fetchCustodySubnetCount* (node: BeaconNode): uint64=
res = DATA_COLUMN_SIDECAR_SUBNET_COUNT.uint64
res

proc fetchCustodyColumnCountFromRemotePeer*(node: BeaconNode, pid: PeerId): uint64 =
# Fetches the custody column count from a remote peer
# if the peer advertises their custody column count
# via the `csc` ENR field. If the peer does NOT, then
# the default value is assume, i.e, CUSTODY_REQUIREMENT
let
eth2node = node.network
peer = eth2node.getPeer(pid)

let enrOpt = peer.enr
if enrOpt.isNone:
debug "Could not get ENR from peer",
peer_id = pid
return 0

else:
let
enr = enrOpt.get
enrFieldOpt = enr.get(enrCustodySubnetCountField, uint64)

if not enrFieldOpt.isOk:
debug "Issue with fetching `csc` field from ENR",
enr = enr
else:
return(enrFieldOpt.get)

proc addDenebMessageHandlers(
node: BeaconNode, forkDigest: ForkDigest, slot: Slot) =
node.addCapellaMessageHandlers(forkDigest, slot)
Expand Down
2 changes: 1 addition & 1 deletion beacon_chain/sync/sync_manager.nim
Original file line number Diff line number Diff line change
Expand Up @@ -548,7 +548,7 @@ proc syncStep[A, B](man: SyncManager[A, B], index: int, peer: A)
withBlck(blck[]):
when consensusFork >= ConsensusFork.Deneb:
if forkyBlck.message.body.blob_kzg_commitments.len > 0:
hasColumns = true
hasColumns = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adds trailing space

break
hasColumns

Expand Down
Loading