[rawo]

Closes #131. Part of the 2.0.0 breaking-change set (labelled critical too).

Stacked on #145 (#135, Postgres TIMESTAMPTZ), which is stacked on #144 (#138, MySQL DATETIME). The new next_attempt_at column uses the same time representation as those PRs. Merge them in order: #144, #145, then this one.

Problem

A retriable failure was retried on the very next poll. With the defaults (interval 1s, max-retries 5), an entry was attempted 6 times in about 6 s and then marked FAILED, which is terminal. So:

• A short downstream outage dead-letters everything in flight.
• Failing entries, always the oldest, are re-claimed first on every tick and delay healthy ones.
• During an outage, every tick hits the failing endpoint with the full batch.

Fix

• next_attempt_at / OutboxEntry.nextAttemptAt: when a PENDING entry is next due.
  • createPending sets it to createdAt.
  • A retriable failure sets it to now + RetryPolicy.backoff(retryNumber).
• RetryPolicy gains initialBackoff (10s), backoffMultiplier (2.0), maxBackoff (10m) and jitter (0.2), all defaulted and @JvmOverloads, plus backoff(retry, random).
  • Delay = initial × multiplier^(n-1), capped, then shortened by up to jitter at random.
  • With the defaults, 5 retries cover about 5 minutes of outage instead of 6 seconds.
• Claim: RouteAwareOutboxStore.claimPending(types, limit, now) returns only rows with next_attempt_at <= now, ordered by (delivery_type, next_attempt_at, id). now comes from the processor's Clock, the same clock that computed the backoff, so database clock skew doesn't matter.
  • The new index on exactly those columns means the scan stops at now. Entries still backing off are never scanned or locked, however large the outage backlog.
  • Ordering by due time also removes the head-of-line blocking.
• Migration 004__retry_backoff (both databases) adds the column, backfills PENDING rows with created_at, and replaces 002's claim index.
  • PostgreSQL: the column is NOT NULL DEFAULT CURRENT_TIMESTAMP, so rows that 1.x instances insert during a rolling deploy stay claimable. This works without a table rewrite on PG 11+. The index goes in a separate runInTransaction:false changeset with CREATE INDEX CONCURRENTLY, dropping any invalid leftover first.
  • MySQL: the column is nullable (an INSTANT add on 8.0.12+), lock_wait_timeout is bounded, and the index is added and dropped INPLACE, LOCK=NONE. 1.x instances are already required to be stopped by fix(mysql)!: store timestamps as UTC DATETIME(6) so the outbox survives 2038 (#138) #144, so the column needs no default.
• Spring: new okapi.processor.initial-backoff, backoff-multiplier, max-backoff and backoff-jitter properties, validated at startup, with configuration metadata.
• Docs: README (retry budget, ordering, properties table, migration list), CHANGELOG under Changed (BREAKING), and UPGRADING.

Breaking

• Retry timing: retries are no longer immediate.
• API:
  • RetryPolicy, OutboxEntry and OutboxProcessorProperties gained parameters, so Kotlin copy/default-argument callers must be recompiled.
  • RouteAwareOutboxStore.claimPending takes now. That interface is new in 2.0, so no released API changes there.
• Schema: new migration 004.

Out of scope

The issue also suggests these. Both are additive, so they can ship in a minor release, and I'll open follow-ups:

• a requeueFailed API or Actuator endpoint;
• limiting retries by elapsed time.

Tests

• RetryPolicyTest:
  • exponential growth and the cap;
  • no overflow at retry 10 000;
  • jitter bounds, using a fixed random value and 1000 random samples;
  • the default retry window;
  • validation;
  • the entry processor scheduling nextAttemptAt.
• RouteAwareOutboxProcessorTest: the processor passes its clock's now to the claim.
• Store contract tests (Postgres and MySQL):
  • a not-yet-due entry isn't claimed until it is due;
  • a retried entry doesn't block a newer one;
  • updateAfterProcessingBatch persists nextAttemptAt;
  • end to end, a real processor and database with a controllable clock don't redeliver before the backoff and do redeliver right after it.
• Migration tests from fix(mysql)!: store timestamps as UTC DATETIME(6) so the outbox survives 2038 (#138) #144 and fix(postgres)!: store timestamps as TIMESTAMPTZ so they no longer depend on the JVM zone (#135) #145: they now also run 004. The Postgres rolling-deploy test inserts 1.x-style rows without the column after 004, and they are claimed.
• Retry-driving end-to-end tests (HttpEndToEndTest, ObservabilityEndToEndTest) now use an explicit zero-backoff policy.
• ./gradlew ktlintCheck test passes locally, and compileJmhKotlin compiles.

🤖 Generated with Claude Code

[rawo]

Reviewed the #146 diff only (on top of #145/#144). The approach is sound: claiming by due time, with an index that stops at now, fixes the dead-lettering and the head-of-line problem.

What I verified. I ran the claim queries against throwaway containers (PostgreSQL 16 and MySQL 8.0.46; 300k/200k rows, 10% PENDING, 80% of those not yet due):

• PostgreSQL, generic prepared plan (plan_cache_mode = force_generic_plan), both the 1-type and 2-type IN forms: Limit -> LockRows -> Index Scan using idx_okapi_outbox_pending_delivery_next_attempt_id, with Index Cond: delivery_type = ANY(...) AND next_attempt_at <= $3. No Sort node, about 5 buffers.
• MySQL with FORCE INDEX: type: range, Using index condition, no filesort. EXPLAIN ANALYZE shows one bounded range per route: (status='PENDING' AND delivery_type='http' AND NULL < next_attempt_at <= now) OR (... 'kafka' ...).

So the index claims in the description hold on both databases. The PG fast-default claim also holds: CURRENT_TIMESTAMP is STABLE, not volatile, so PG 11+ stores it in attmissingval.

The inline comments cover the specific issues. Cross-cutting ones:

1. should-fix, tests don't prove the migration/rolling-deploy claims. The PR description says the Postgres rolling-deploy test "inserts 1.x-style rows without the column after 004, and they are claimed". PostgresTimestamptzMigrationTest does insert after the full migrate(url), but it asserts with the legacy store.claimPending(10), which ignores next_attempt_at. If the DEFAULT were missing or wrong, the test would still pass. Nothing tests:

   • the 004 backfill: pre-existing PENDING rows get next_attempt_at = created_at and keep their order under the route-aware claim;
   • the MySQL NULL → created_at fallback in toOutboxEntry;
   • that a 1.x-style insert after 004 is returned by claimPending(types, limit, Instant.now()).

   Suggest switching those assertions to the route-aware claim and adding a pre-004 PENDING row to both migration tests.

2. nit, no Spring test for the four new properties. Nothing checks binding (e.g. okapi.processor.initial-backoff=1s reaching the RetryPolicy) or that an invalid value fails startup with the property name. OutboxProcessorAutoConfigurationTest already has the setup for this.

3. question, clock sources. The description says "now comes from the processor's Clock ... so database clock skew doesn't matter". That is true for retries. A new row's due time, though, is createdAt from the publishing instance's clock. On PG, a row inserted by 1.x during the rolling window gets the DB's CURRENT_TIMESTAMP. So skew between a publisher (or the DB) and a processor now delays brand-new messages by that skew: a publisher 30 s ahead means its messages wait about 30 s. Before this PR, skew only affected ordering. With NTP this is small. Is it worth a sentence in README's ordering/guarantees section, or using <= now + small tolerance? Not blocking.

[rawo]

should-fix: the backoff is measured from before delivery, so a slow failure uses up most or all of it.

In processBatch, now is read before deliverer.deliverBatch(entries) (line 34), so nextAttemptAt = batchStart + backoff. A slow failure is the typical shape of a real outage. For example, HttpMessageDeliverer has a 30 s exchange timeout and a 10 s connect timeout.

• The 10 s and 20 s backoffs (with jitter, 8 to 20 s) have already passed by the time updateAfterProcessingBatch commits.
• Those entries are claimed again on the very next tick.
• During an outage with timeouts, the first two retries are still back-to-back, which is the behaviour No retry backoff: default config dead-letters every pending message after ~6s of downstream outage #131 reports.

The single-entry process() path reads the clock after deliver(), so the two paths also disagree.

Suggestion: in processBatch, take a second clock.instant() after deliverBatch returns and use it as the base for nextAttemptAt. lastAttempt/updatedAt can keep the start time if that is intended. A unit test with a clock that advances inside deliverBatch would pin this down.

[rawo]

should-fix: the backfill runs while the table's ACCESS EXCLUSIVE lock is held.

The ALTER TABLE on line 13 takes ACCESS EXCLUSIVE, and PostgreSQL holds it until the changeset's transaction commits. This UPDATE runs in the same transaction, so the lock covers the whole backfill, not a "brief" moment as line 10 and UPGRADING.md say. Every publish(), and every read of okapi_outbox (1.x workers, lag metrics), waits until all PENDING rows are rewritten. The PENDING backlog is largest right after an outage, which is exactly when people are likely to upgrade for this fix, and every PENDING row gets a new tuple plus index entries.

Suggestion: move the UPDATE into its own changeset (e.g. outbox:004-backfill) after the column add.

• In its own transaction it takes only row locks, so publish() (inserts) is not blocked, and 1.x SKIP LOCKED claims just skip rows the backfill is touching.
• It stays idempotent on re-run.
• Optionally, batch it by id range for very large backlogs.
• Then fix the comment on line 10 and the UPGRADING wording.

[rawo]

should-fix: the changeset cannot be re-run after a partial failure.

MySQL DDL commits implicitly, so the steps in this changeset are not atomic.

• Suppose the ADD INDEX (line 18) or DROP INDEX (line 22) fails. The likeliest cause is the 30 s lock_wait_timeout added here: the INPLACE build needs a brief exclusive MDL and waits behind any open transaction on the table.
• By then the column already exists and the backfill is committed, but Liquibase has not recorded outbox:004.
• On the next start this line fails with Duplicate column name 'next_attempt_at', and every start fails after that until someone fixes the schema by hand.

The comment in 003 says a timeout means "the migration fails and runs again on the next start". That holds for 003 (a single statement) but not for this changeset. MySQL has no ADD COLUMN IF NOT EXISTS.

Suggestion: split this into one changeset per DDL statement (column, backfill, add index, drop index), each with a Liquibase precondition, for example:

• --precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM information_schema.columns WHERE table_schema = DATABASE() AND table_name = 'okapi_outbox' AND column_name = 'next_attempt_at'
• an onFail:MARK_RAN precondition on information_schema.statistics for the two index statements.

Minor: as in 003, lock_wait_timeout is only restored on success. The 003 comment documents that; this file doesn't.

[rawo]

should-fix: a manually requeued pre-004 FAILED row can never be claimed on MySQL.

Only PENDING rows are backfilled, so FAILED (and DELIVERED) rows written before 004 keep next_attempt_at = NULL. There is no requeue API yet: it is deferred to a follow-up, and README says an upgrade does not retry existing FAILED rows. The usual workaround is therefore UPDATE okapi_outbox SET status = 'PENDING', retries = 0 WHERE ....

• On PostgreSQL that works, because those rows got the column default.
• On MySQL next_attempt_at <= ? is never true for NULL, so the row stays PENDING forever.
• toOutboxEntry's ?: created_at fallback hides this, and nothing logs it.

Suggestion: backfill WHERE status IN ('PENDING', 'FAILED'). FAILED rows are usually few, and DELIVERED rows get purged, so the extra cost is small. Alternatively, or as well, add an UPGRADING note that a manual requeue must also set next_attempt_at. The same advice would be useful generally, for example SET next_attempt_at = CURRENT_TIMESTAMP(6) on MySQL (UTC session), since requeued rows keep their old due time.

[rawo]

should-fix (docs): this paragraph is now stale.

It still says to apply 002 before starting updated workers and that MySQL claims use "the new index". After this PR:

• MysqlOutboxStore uses FORCE INDEX (idx_okapi_outbox_status_delivery_next_attempt_id) and needs the next_attempt_at column, so 004 must be applied before updated workers start, or every claim fails.
• PostgreSQL's 004 claim-index changeset is also CREATE INDEX CONCURRENTLY with the same caveats about waiting for transactions.

Suggest: "apply all migrations up to 004 ...", and mention that the CONCURRENTLY caveats apply to both 002 and 004.

[rawo]

nit: two edge cases in backoff().

• toMillis() truncates, so a sub-millisecond initialBackoff silently becomes 0.
• More importantly, maxBackoff has no upper bound. Duration.toMillis() throws ArithmeticException above about 292M years. A very large maxBackoff (e.g. from a typo in a Spring property) passes validation, then backoff() throws inside processBatch on the first retriable failure. The batch transaction rolls back and the same entries fail again every tick. now + backoff can also overflow Instant or exceed the DB column range well before that.

Suggest validating maxBackoff against a sane bound in init, e.g. <= Duration.ofDays(365), or clamping. Also, backoffMultiplier = +Infinity passes >= 1.0, and with initialBackoff = 0 it gives 0 * Inf = NaN. That happens to come out as 0 ms, but require(backoffMultiplier.isFinite()) would be cleaner.

[rawo]

nit (API docs): Java callers lose the 11-argument constructor.

OutboxEntry's constructor has no @JvmOverloads, so the 11-argument JVM constructor is gone and Java callers of new OutboxEntry(...) get a compile error. The CHANGELOG only says Kotlin copy/default-argument callers must recompile. Either add @JvmOverloads constructor here, as RetryPolicy and OutboxProcessorProperties have, or mention Java constructor callers in the CHANGELOG.

[rawo]

Addressed the review in 56b79f1:

1. Backoff counted from before delivery. Fixed: processBatch now reads the clock after deliverBatch, as process() already did. New test: a deliverer that takes 30 s gives nextAttemptAt = start + 30 s + 10 s.
2. Postgres backfill held the ACCESS EXCLUSIVE lock. Fixed: the backfill is now a separate changeset (004-backfill), so the exclusive lock covers only the catalog-only ADD COLUMN. The SQL comment and UPGRADING were corrected.
3. MySQL 004 couldn't be re-run after a partial failure. Fixed: each DDL step is now its own changeset (004, 004-backfill, 004-claim-index, 004-drop-created-index) with an information_schema precondition and onFail:MARK_RAN, so a run that stopped half-way resumes.
4. FAILED rows left with NULL. Fixed: the backfill now covers every non-DELIVERED row. UPGRADING documents the manual requeue recipe, which includes next_attempt_at. The MySQL migration test now requeues a pre-migration FAILED row by hand and claims it through the route-aware claim.
5. README "apply 002" text. Fixed: it now says to apply all migrations up to 004 before starting updated workers, and that 004's index changeset on Postgres uses CONCURRENTLY.
6. Tests used the legacy claim. Fixed: both migration tests now go through the route-aware claim:
   • Postgres upgrade path: the backfilled nextAttemptAt equals createdAt.
   • Postgres rolling deploy: a 1.x insert after 004 is claimable only thanks to the column DEFAULT.
   • MySQL: backfill, plus the FAILED requeue from point 4.
7. No Spring property tests. Added: the four properties bind, defaults come from RetryPolicy, and invalid values (multiplier < 1, jitter > 1, initial > max, max > 365 days) fail startup.
8. Clock skew. You're right, my wording was wrong. A new row is due at the publisher's createdAt, so a publisher clock running ahead delays its messages by the difference. Documented in UPGRADING with an NTP note. The PR description's "doesn't matter" claim is withdrawn here.
9. RetryPolicy edge cases. Backoff is now computed in nanoseconds, so sub-millisecond values are kept. maxBackoff is capped at 365 days, and the multiplier must be finite. The same checks apply to the Spring properties. Tests cover all three.
10. Java constructor. Fixed: the OutboxEntry constructor is now @JvmOverloads, so Java callers of the 11-argument form still compile. Noted in the CHANGELOG.

I also hit and fixed a Liquibase 5 parse error: a comment line starting -- changeset's was being read as a directive.

./gradlew ktlintCheck test passes.