feat(shard-distributor): add SpectatorPeerChooser for shard-aware routing #7478
+367
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jakobht
requested review from
3vilhamster,
Shaddoll,
davidporter-id-au,
demirkayaender,
dkrotx,
neil-xie,
sankari165,
shijiesheng and
taylanisikdemir
as code owners
jakobht
force-pushed
the
add-spectator-peer-chooser
branch
4 times, most recently
from
642ad52 to
1597904
Compare
jakobht
force-pushed
the
add-spectator-peer-chooser
branch
2 times, most recently
from
f4bb653 to
85017c4
Compare
dkrotx
reviewed
Nov 28, 2025
service/sharddistributor/client/spectatorclient/peer_chooser.go
Outdated
Show resolved
Hide resolved
service/sharddistributor/client/spectatorclient/peer_chooser.go
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| // Extract GRPC address from owner metadata | ||
| grpcAddress, ok := owner.Metadata[grpcAddressMetadataKey] |
Member
There was a problem hiding this comment.
nit: a bit unexpected to see such specifics as grpcAddressMetadataKey here.
Member
Author
There was a problem hiding this comment.
Yeah, I would like better separation. We need to address here, so I decided to do it like this, but it does feel like mixing application and network level.
service/sharddistributor/client/spectatorclient/peer_chooser.go
Outdated
Show resolved
Hide resolved
jakobht
force-pushed
the
add-spectator-peer-chooser
branch
from
89bb17f to
1e6cc8a
Compare
Implement a YARPC peer chooser that routes requests to the correct
executor based on shard ownership. This is the shard distributor
equivalent of Cadence's RingpopPeerChooser.
Flow:
1. Client calls RPC with yarpc.WithShardKey("shard-key")
2. Chooser queries Spectator for shard owner
3. Extracts grpc_address from owner metadata
4. Creates/reuses peer for that address
5. Returns peer to YARPC for connection
The peer chooser maintains a cache of peers and handles concurrent
access safely. It uses the x-shard-distributor-namespace header to
determine which namespace's spectator to query.
Dependencies:
- Requires spectator GetShardOwner to return metadata (see previous commit)
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
The tests have dependency issues with mock generation that need to be resolved separately. The peer chooser implementation is complete and functional. Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Tests cover: - Success path with peer creation - Peer reuse on subsequent calls - Error cases (missing shard key, namespace header, spectator not found) - Lifecycle methods (Start, Stop, IsRunning) - SetSpectators method Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
…peer creation and mutex lock and unlock Signed-off-by: Jakob Haahr Taankvist <[email protected]>
jakobht
force-pushed
the
add-spectator-peer-chooser
branch
from
1e6cc8a to
83f485a
Compare
dkrotx
approved these changes
Dec 1, 2025
jakobht
added a commit
that referenced
this pull request
Dec 2, 2025
…ip verification (#7487) Depends on #7475 and #7478 being merged **What changed?** Added canary pinger component that periodically sends ping requests to shard owners to verify executor-to-executor communication and shard ownership. **Why?** The canary pinger provides active monitoring of the shard distributor's routing and ownership mechanisms by: - Periodically selecting random shards and pinging their owners - Verifying that the pinged executor owns the shard - Detecting communication failures between executors This is part of the canary ping/pong implementation that validates end-to-end executor-to-executor gRPC communication. **How did you test it?** Unit tests **Potential risks** **Release notes** **Documentation Changes** --------- Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR depends on #7476 (spectator return metadata) being merged first.
What changed?
Added
SpectatorPeerChooserthat implements YARPC's peer.Chooser interface to route requests to the correct executor based on shard ownership.Why?
Enable executor-to-executor communication in the shard distributor canary system. The peer chooser queries the Spectator to find which executor owns a shard, then routes requests to that executor's gRPC address.
How did you test it?
Unit tests
Potential risks
None
Release notes
Documentation Changes