Store startup fixes #5926
Merged
Store startup fixes #5926
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lutter
force-pushed
the
lutter/sharded
branch
12 times, most recently
from
6bc5d84 to
fc0ae44
Compare
incrypto32
approved these changes
Apr 7, 2025
Comment on lines
+1654
to
+1661
| // Everything here happens under the migration lock. Anything called | ||
| // from here should not try to get that lock, otherwise the process | ||
| // will deadlock | ||
| debug!(self.logger, "Waiting for migration lock"); | ||
| let res = with_migration_lock(&mut pconn, |_| async { | ||
| debug!(self.logger, "Migration lock acquired"); | ||
|
|
||
| // While we were waiting for the migration lock, another thread |
It should be up to the operator if they use it or not, and when they want to reset it
The current database setup code was inerently racy when several nodes were starting up as it relied on piecemeal locking of individual steps. This change completely revamps the strategy we use: setup now takes a lock on the primary, so that only one node at a time will run the setup code.
Before, PoolState was just an enum and code all over the place dealt with its interior mutability. Now, we encapsulate that to simplify code using the PoolState
Instead of dealing with disabled shards (shards that have a pool size of 0 configured), filter those shards out on startup and warn about them. The end effect is that for that configuration, users will get an error of 'unkown shard' rather than 'shard disabled'. Since configuring a shard to have no connections is kinda pathological, and leads to an error when it is used either way, the code simplification is worth the slightly less helpful error message. Removing the 'disabled' state from pools has ripple effects to quite a few other places, simplifying them a bit
With the previous code, we would run setup initially when creating all pools, but they would not be marked as set up. On the first access to the pool we would try to run setup again, which is not needed. This change makes it so that we remember that we ran setup successfully when pools are created
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With multiple nodes, they could race each other and cause database errors.
I've reworked the startup code to be (a) race-proof (I hope) and (b) easier to follow. Any node that wants to run database setup now gets a lock on the primary, and runs all code needed for setup while holding that lock. That way, nodes can't interfere with each other. In the common case, where there are no database changes, be it because of migrations, configuration changes, or code changes that map different tables, the time during which any node holds the lock is very brief. This could be further optimized but let's first see how this performs in practice.