branch-3.1: [fix](load) Fix the idempotence of the createPartition RPC, ensure consistent tablet replica information in one txn #57198 #58703
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nsistent tablet replica information in one txn (#57198) Problem Summary: During the load of a auto partition table, we will meet this case: let rpc(2025-10-21) is `create partition rpc (need create partition : "2025-10-21" )` sink1 -----------------------------rpc(2025-10-21)------------------------------------------> FE sink2 -----------------------------rpc(2025-10-21)------------------------------------------> FE The FE's processing of these two RPCs is not atomic within a single import transaction, so we need to ensure the idempotence of this RPC. But, if the FE processes the RPC from sink1 first, creating partition "2025-10-21", and then creates several tablets under it while selecting BEs for them. Assuming tablet1's replica is assigned to [BE-1]. ----> This time, a tablet migration occurs (for load balancing), causing tablet1 move from [BE1] to [BE2]. When the FE processes the second RPC, it returns the post-migration information, resulting in inconsistent tablet1 replica distribution information observed by the two RPC initiators. This indicates that the createPartition RPC is not idempotent. Fix this case： When the first RPC creates a partition and selects BE1 as the replica for tablet1, we record this information on a per-transaction basis. In the next RPC, we can simply return the recorded information.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Contributor
|
rerun buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
compile failed |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #57198