-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DEPS: upgrade Raft to 72d62a1 #105578
DEPS: upgrade Raft to 72d62a1 #105578
Conversation
For some reason, etcd-io/raft#77 breaks leader removals. Should've known better than to make opportunistic cleanups, I suppose. Not sure why it breaks yet, will dig into it tomorrow. |
I think the problem is that the leader steps down before we complete the joint quorum change:
The leader is now a learner, but it needs to propose and commit the conf change that actually removes it from the range, otherwise noone else will campaign for leadership. |
The problem is a mismatch between Raft and CRDB. In Raft, the leader steps down once it become a learner in a joint config. In CRDB, we don't elect a new leader until the old leader is fully removed from the range -- so when the leader becomes a learner and steps down, the range remains leaderless. This patch fixes it: --- a/pkg/kv/kvserver/client_raft_test.go
+++ b/pkg/kv/kvserver/client_raft_test.go
diff --git a/pkg/kv/kvserver/replica_raft.go b/pkg/kv/kvserver/replica_raft.go
index d33eece0989..48b1aed4727 100644
--- a/pkg/kv/kvserver/replica_raft.go
+++ b/pkg/kv/kvserver/replica_raft.go
@@ -2570,7 +2570,7 @@ func shouldCampaignAfterConfChange(
// don't expect to hit this, but let's be defensive.
return false
}
- if _, ok := desc.GetReplicaDescriptorByID(roachpb.ReplicaID(raftStatus.Lead)); ok {
+ if r, ok := desc.GetReplicaDescriptorByID(roachpb.ReplicaID(raftStatus.Lead)); ok && r.IsAnyVoter() {
// The leader is still in the descriptor.
return false
} Unfortunately, this will break backwards compatibility with 23.1 nodes. We have a few options:
The Raft thesis describes leader removal in section 4.2.2, we should probably just do what's specified there, with an option to enable/disable it, for backwards compatibility and migrations. |
Making it opt-in in etcd-io/raft#79 |
115bd01
to
362fa76
Compare
Epic: none Release note: None
This patch campaigns when the leader demotes itself to a learner during an atomic conf change, instead of waiting until it is completely removed from the range. Relying on the old leader to commit the final conf change while it's a learner appears unfortunate, and is not compatible with an upcoming etcd/raft change that makes the learner step down in this case. This is backwards compatible with 23.1, because the same follower replica is responsible for campaigning, and it does not matter if it campaigns during the demotion or removal -- in particular because it forces an immediate election via `forceCampaignLocked()` which immediately bumps the term. Epic: none Release note: None
This causes the Raft leader to step down when it removes itself from the range or demotes itself to a learner. This doesn't make any difference functionally, since we're careful to no longer tick the replica or otherwise use it, but in principle it could cause a leader that was no longer part of the range to continue to assert leadership, stalling the range since it wouldn't be able to propose anything. Stepping down appears safer and cleaner. Epic: none Release note: None
Closing in favor of #107203. Raft has already been bumped. |
Makes Raft leaders step down when removing themselves from the range.
Epic: none
Release note: None