DRIVERS-3239: Add exponential backoff to operation retry loop for server overloaded errors #1862

baileympearson · 2025-12-02T17:15:50Z

Overview

This PR adds support for a new class of errors (SystemOverloadedError) to drivers' operation retry logic, as outlined in the design document.

Additionally, it includes a new argument to the MongoDB handshake (also defined in the design document).

Python will be second implementer.
Node implementation: mongodb/node-mongodb-native#4806

Testing

The testing strategy is two-fold:

Building off of Ezra's work to generate unified tests for retryable handshake errors, this PR generates unified tests to confirm that:
- operations are retried using the new SystemOverloadedError label
- operations are retried no more than 5 (current MAX_ATTEMPTS, as defined in the spec) times
Following Iris's work in DRIVERS-1934: withTransaction API retries too frequently #1851, this PR adds a prose test that ensures drivers apply exponential backoff in the retryability loop.
Update changelog.
Test changes in at least one language driver.
Test these changes against all server versions and topologies (including standalone, replica set, and sharded
clusters).

- add prose test - add assertions on the number of retries for maxAttempts tests - don't run clientBulkWrite tests on <8.0 servers

source/logging/logging.md

source/client-backpressure/client-backpressure.md

source/retryable-reads/retryable-reads.md

source/retryable-writes/retryable-writes.md

source/client-backpressure/client-backpressure.md

blink1073 · 2025-12-03T16:31:19Z

It looks like you also need to bump the schema version:

source/client-backpressure/tests/backpressure-retry-loop.yml invalid
[
  {
    instancePath: '/tests/0/operations/3/expectError',
    schemaPath: '#/definitions/expectedError/type',
    keyword: 'type',
    params: { type: 'object' },
    message: 'must be object'
  }
]
 using schema v1.3
source/client-backpressure/tests/backpressure-retry-max-attempts.yml invalid
[
  {
    instancePath: '/tests/0/operations/1/expectError',
    schemaPath: '#/definitions/expectedError/type',
    keyword: 'type',
    params: { type: 'object' },
    message: 'must be object'
  }
]
 using schema v1.3source/client-backpressure/tests/backpressure-retry-loop.yml invalid
[
  {
    instancePath: '/tests/0/operations/3/expectError',
    schemaPath: '#/definitions/expectedError/type',
    keyword: 'type',
    params: { type: 'object' },
    message: 'must be object'
  }
]
 using schema v1.3
source/client-backpressure/tests/backpressure-retry-max-attempts.yml invalid
[
  {
    instancePath: '/tests/0/operations/1/expectError',
    schemaPath: '#/definitions/expectedError/type',
    keyword: 'type',
    params: { type: 'object' },
    message: 'must be object'
  }
]
 using schema v1.3

blink1073 · 2025-12-03T21:05:18Z

WIP Python implementation: mongodb/mongo-python-driver#2635

blink1073 · 2025-12-03T23:11:38Z

All unified and prose tests are passing in the Python implementation.

Edit: we're still failing one unified test, "client.clientBulkWrite retries using operation loop", investigating...

Edit 2: we're all good now

jyemin

I only reviewed the specification changes, not the pseudocode or tests. Those are best reviewed by implementers.

jyemin · 2025-12-10T19:26:49Z

source/client-backpressure/client-backpressure.md

+    - This intentionally changes the behavior of CSOT which otherwise would retry an unlimited number of times within the
+        timeout to avoid retry storms.
+5. If the previous error includes the `SystemOverloadedError` label, the client MUST apply exponential backoff according
+    to according to the following formula: `delayMS = j * min(maxBackoff, baseBackoff * 2^i)`


Suggested change

to according to the following formula: `delayMS = j * min(maxBackoff, baseBackoff * 2^i)`

to the following formula: `delayMS = j * min(maxBackoff, baseBackoff * 2^i)`

jyemin · 2025-12-10T19:33:11Z

source/client-backpressure/client-backpressure.md

+
+This specification expands the driver's retry ability to all commands, including those not currently considered
+retryable such as updateMany, create collection, getMore, and generic runCommand. The new command execution method obeys
+the following rules:


Since the rules include all the deposits into the token bucket, consider adding withdrawals as well.

jyemin · 2025-12-10T19:36:28Z

source/client-backpressure/client-backpressure.md

+
+## Q&A
+
+TODO


Anything to add here, or just remove?

Nothing yet .. I'll remove it for now. It can always be added back if there are items worth mentioning here.

jyemin · 2025-12-10T19:37:07Z

source/client-backpressure/client-backpressure.md

+
+## Changelog
+
+- 2025-XX-XX: Initial version.


Not sure how we handle the date... Is there an automation for this?

Not that I know of. Usually the spec author fills it out before merging

I'll just leave this thread open to remind myself to add changelog dates before merging once all changes are completed.

source/client-backpressure/tests/README.md

source/client-backpressure/client-backpressure.md

Jibola · 2025-12-10T20:51:37Z

source/client-backpressure/client-backpressure.md

+to identify clients which do and do not support backpressure. Currently, this flag is unused but in the future the
+server may offer different rate limiting behavior for clients that do not support backpressure.
+
+##### Implementation notes


Suggested change

##### Implementation notes

#### Implementation notes

source/retryable-writes/retryable-writes.md

source/transactions/tests/unified/backpressure-retryable-commit.yml

source/transactions/tests/unified/backpressure-retryable-abort.yml

source/transactions/tests/unified/backpressure-retryable-commit.yml

source/transactions/tests/unified/backpressure-retryable-reads.yml

source/transactions/tests/unified/backpressure-retryable-writes.yml

jmikola · 2025-12-18T18:38:25Z

source/transactions/transactions.md


 An error considered retryable by the [Retryable Writes Specification](../retryable-writes/retryable-writes.md).

+#### Backpressure Error


This spec should also get a changelog entry.

baileympearson

submitting comments

source/client-backpressure/tests/README.md

baileympearson · 2025-12-18T22:25:45Z

source/client-backpressure/tests/backpressure-retry-loop.yml.template

+    operations:
+      -
+        object: *utilCollection
+        name: deleteMany


Sorry, missed this one. I like your suggestion - done.

baileympearson · 2025-12-18T22:27:22Z

source/client-backpressure/client-backpressure.md

+    - This intentionally changes the behavior of CSOT which otherwise would retry an unlimited number of times within the
+        timeout to avoid retry storms.
+5. If the previous error includes the `SystemOverloadedError` label, the client MUST apply exponential backoff according
+    to according to the following formula: `delayMS = j * min(maxBackoff, baseBackoff * 2^i)`


baileympearson · 2025-12-18T22:27:26Z

source/client-backpressure/client-backpressure.md

+
+This specification expands the driver's retry ability to all commands, including those not currently considered
+retryable such as updateMany, create collection, getMore, and generic runCommand. The new command execution method obeys
+the following rules:


source/client-backpressure/client-backpressure.md

baileympearson · 2025-12-18T22:47:43Z

source/client-backpressure/client-backpressure.md

+The following pseudocode describes the overload retry policy:
+
+```python
+BASE_BACKOFF = 0.1


I chose to leave it as-is; as the pseudocode is written in python and follows python conventions. Let me know if the clarified pseudocde works for you

source/client-backpressure/client-backpressure.md

baileympearson · 2025-12-18T22:48:29Z

source/client-backpressure/tests/README.md

+
+## Changelog
+
+- 2025-XX-XX: Initial version.


I removed the changelog

baileympearson · 2025-12-18T22:49:36Z

source/client-backpressure/tests/README.md

+            }
+        ```
+
+    3. Execute the following command. Expect that the command errors. Measure the duration of the command execution.


I've reworked the prose to be explicit about what the command is.

matthewdale

Looks good! 👍

sanych-sun · 2025-12-19T00:21:44Z

source/client-backpressure/client-backpressure.md

+5. A retry attempt consumes 1 token from the token bucket.
+6. If the request is eligible for retry (as outlined in step 4), the client MUST apply exponential backoff according to
+    the following formula: `delayMS = j * min(maxBackoff, baseBackoff * 2^i)`
+    - `i` is the retry attempt number (starting with 0 for the first retry).


Why we decided to have retry number start we 0? It makes the requirements confusing. Why don't we start with 1 and in fact we will have the same formula as for withTransaction:

delayMS = j * min(maxBackoff, baseBackoff * 2^(i-1))

With the only difference here we have 2 as the base for pow function, in convinientTransaction API we have 1.5

Here is the formula from withTransaction:
jitter * min(BACKOFF_INITIAL * 1.5 ** (transactionAttempt - 1), BACKOFF_MAX)

Where transactionAttempt started with 0 and is being incremented AFTER the delay, but before executing the callback attempt. Which is also confusing... but in C# implementation we wait AFTER the attempt so it's more natural.

The phrasing, including "Retries start at 0", was just taken from the design. There's no need to keep it this way if it causes confusion.

I can adjust the phrasing to more closely align with the transaction spec, if that's preferable?

Can we use the following formula?
delayMS = j * min(maxBackoff, baseBackoff * 2^(i-1))

or

we can keep the formula as is, but adjust the baseBackoff :
delayMS = j * min(maxBackoff, baseBackoff * 2^i)
where baseBackoff is 50 instead of 100.

It produces the same results, but starts i with 1.

I think it's fine to use any of those three formulas in individual driver implementations if it increases readability so long as they result in the same outputs. As far as reducing confusion, I can say it didn't substantially change my understanding of the formula.

+1 to Bailey changing the phrasing, which I think would suffice.

…RS-3239

source/client-backpressure/client-backpressure.md

stIncMale · 2025-12-25T00:01:24Z

source/client-backpressure/client-backpressure.md

+# Note: the values below have been scaled down by a factor of 1000 because
+# Python's sleep API takes a duration in seconds, not milliseconds.
+BASE_BACKOFF = 0.1 # 100ms
+MAX_BACKOFF = 10   # 10s


The 100ms comment is helpful, because one can use it to find the same value in the prose part of the specification (though absolutely nothing prevents us from simply writing BASE_BACKOFF = 100ms).

However, the 10s comment is not helpful, because the prose part says 10000ms, and a reader won't find 10000ms in the pseudocode.

Suggested change

MAX_BACKOFF = 10 # 10s

MAX_BACKOFF = 10 # 10000ms

baileympearson added 8 commits December 1, 2025 13:44

initial commit

588f1f2

new files

e467f5b

add tests for handshake changes

d55fdb9

add generated tests

8e74b41

test fixes and add prose test

072b453

- add prose test - add assertions on the number of retries for maxAttempts tests - don't run clientBulkWrite tests on <8.0 servers

fix run on requirements

52e2a35

fix run on requirements?

391c951

fix CI

92501c0

baileympearson commented Dec 2, 2025

View reviewed changes

source/logging/logging.md Outdated Show resolved Hide resolved

baileympearson commented Dec 2, 2025

View reviewed changes

source/client-backpressure/client-backpressure.md Outdated Show resolved Hide resolved

baileympearson marked this pull request as ready for review December 2, 2025 18:59

baileympearson requested review from a team as code owners December 2, 2025 18:59

baileympearson requested review from jmikola and jyemin and removed request for a team December 2, 2025 18:59

blink1073 reviewed Dec 3, 2025

View reviewed changes

source/retryable-reads/retryable-reads.md Outdated Show resolved Hide resolved

source/retryable-writes/retryable-writes.md Outdated Show resolved Hide resolved

source/client-backpressure/client-backpressure.md Outdated Show resolved Hide resolved

baileympearson added 3 commits December 3, 2025 11:36

comments

0fdef39

Fix broken unified tests

82acab8

fix UTR linting failures

b3a7b6c

blink1073 mentioned this pull request Dec 3, 2025

PYTHON-5528 & PYTHON-5651 Add exponential backoff to operation retry loop for server overloaded errors mongodb/mongo-python-driver#2635

Draft

11 tasks

remove broken deleteMany() from unified tests

60a87b8

add backwards compat section

399a56b

jyemin requested changes Dec 10, 2025

View reviewed changes

Jibola requested changes Dec 10, 2025

View reviewed changes

stIncMale reviewed Dec 11, 2025

View reviewed changes

source/retryable-writes/retryable-writes.md Show resolved Hide resolved

blink1073 requested a review from a team as a code owner December 17, 2025 12:41

blink1073 requested review from durran and removed request for a team December 17, 2025 12:41

blink1073 added 5 commits December 17, 2025 10:18

update transaction logic and add more tests

6912e45

verify commitTransaction fails after 5 backoff attempts

88e6067

clean up transactions spec

2019678

update test names

d3ce32b

address writeconcern on retries

de7e862

baileympearson commented Dec 18, 2025

View reviewed changes

add retryable get more tests

ab85a5b

jmikola reviewed Dec 18, 2025

View reviewed changes

baileympearson and others added 4 commits December 18, 2025 12:12

transaction test fixes

eb10ddb

deduplicate ids

2b90697

update transaction writeconcern logic and add changelog entry

0abf373

last few comments

d07d49e

baileympearson requested review from jmikola, matthewdale and vbabanin December 18, 2025 22:50

baileympearson commented Dec 18, 2025

View reviewed changes

matthewdale approved these changes Dec 18, 2025

View reviewed changes

sanych-sun reviewed Dec 19, 2025

View reviewed changes

sanych-sun mentioned this pull request Dec 19, 2025

CSHARP-5712: withTransaction API retries too frequently mongodb/mongo-csharp-driver#1841

Open

blink1073 and others added 6 commits December 19, 2025 06:21

add runOnRequirement for getMore tests

747c18c

updated formula

be27bc0

Merge branch 'master' of github.com:mongodb/specifications into DRIVE…

92e479a

…RS-3239

lint

b674f13

address review

03065ad

fix link

1b7f6df

Jibola approved these changes Dec 23, 2025

View reviewed changes

source/client-backpressure/client-backpressure.md Show resolved Hide resolved

stIncMale reviewed Dec 25, 2025

View reviewed changes

	to according to the following formula: `delayMS = j * min(maxBackoff, baseBackoff * 2^i)`
	to the following formula: `delayMS = j * min(maxBackoff, baseBackoff * 2^i)`


		An error considered retryable by the [Retryable Writes Specification](../retryable-writes/retryable-writes.md).

		#### Backpressure Error


		## Q&A

		TODO

DRIVERS-3239: Add exponential backoff to operation retry loop for server overloaded errors #1862

Are you sure you want to change the base?

DRIVERS-3239: Add exponential backoff to operation retry loop for server overloaded errors #1862

Uh oh!

Conversation

baileympearson commented Dec 2, 2025 • edited by blink1073 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blink1073 commented Dec 3, 2025

Uh oh!

blink1073 commented Dec 3, 2025

Uh oh!

blink1073 commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jyemin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

baileympearson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthewdale left a comment

Choose a reason for hiding this comment

Uh oh!

sanych-sun Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

baileympearson commented Dec 2, 2025 •

edited by blink1073

Loading

blink1073 commented Dec 3, 2025 •

edited

Loading

sanych-sun Dec 19, 2025 •

edited

Loading

sanych-sun Dec 19, 2025 •

edited

Loading

sanych-sun Dec 19, 2025 •

edited

Loading

stIncMale Dec 25, 2025 •

edited

Loading