feat(shard-distributor): Add canary pinger for periodic shard ownership verification #7487
+279
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jakobht
requested review from
3vilhamster,
Shaddoll,
davidporter-id-au,
demirkayaender,
dkrotx,
neil-xie,
sankari165,
shijiesheng and
taylanisikdemir
as code owners
jakobht
changed the title
jakobht
force-pushed
the
add-canary-pinger-new
branch
from
34d56e5 to
ddf0d74
Compare
jakobht
changed the title
jakobht
force-pushed
the
add-canary-pinger-new
branch
4 times, most recently
from
41c171c to
1c77d48
Compare
Implement the client-side pinger that periodically pings random shard owners to verify: 1. Executors can route to each other based on shard ownership 2. Shard ownership information is accurate 3. The shard distributor is functioning correctly The pinger: - Selects random shards at regular intervals (1s with 10% jitter) - Sends ping requests to the executor owning each shard - Validates that the receiving executor actually owns the shard - Logs warnings when ownership is incorrect Dependencies: - Requires ShardDistributorExecutorCanaryAPI proto and client - Will use SpectatorPeerChooser for routing (wired in later commit) Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
jakobht
force-pushed
the
add-canary-pinger-new
branch
from
1c77d48 to
30fc04e
Compare
eleonoradgr
reviewed
Dec 2, 2025
| } | ||
|
|
||
| // Verify response | ||
| if !response.GetOwnsShard() { |
Contributor
There was a problem hiding this comment.
when do we expect this? is it in case spectator client in the peerchooser has outdated info?
Member
Author
There was a problem hiding this comment.
yes - it's also a test. This should rarely happen, so if canary sees this a lot we have problems.
eleonoradgr
approved these changes
Dec 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on #7475 and #7478 being merged
What changed?
Added canary pinger component that periodically sends ping requests to shard owners to verify executor-to-executor communication and shard ownership.
Why?
The canary pinger provides active monitoring of the shard distributor's routing and ownership mechanisms by:
This is part of the canary ping/pong implementation that validates end-to-end executor-to-executor gRPC communication.
How did you test it?
Unit tests
Potential risks
Release notes
Documentation Changes