Introduce lease-based fencing for task attempts and heartbeats #111
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| or ( | ||
| state = $3 | ||
| -- Has heartbeat stalled? | ||
| and last_heartbeat_at < now() - heartbeat | ||
| and (lease_expires_at < now() or lease_expires_at is null) | ||
| -- Are there remaining retries? |
There was a problem hiding this comment.
The new staleness predicate treats lease_expires_at is null as immediately stale, but the migration only adds the column without backfilling existing in-progress rows, so any task that was already in_progress at deploy time will be re-dequeued right away even if a worker is still running it. That can lead to duplicate processing and inconsistent attempt histories during rolling upgrades. Consider either backfilling lease_expires_at for existing in-progress tasks or falling back to last_heartbeat_at when lease_expires_at is null.
Useful? React with 👍 / 👎.
This change adds lease tokens/expiry to task and attempt rows, uses them to fence state transitions and heartbeats, and updates staleness detection to rely on lease expiry. It prevents stale or resumed workers from mutating task state after a new attempt has taken over, improving safety under pauses or crashes.
maxcountryman
force-pushed
the
feat/lease-based-fencing-attempts-heartbeats
branch
from
f69ac0e to
dbc9b8a
Compare
This change adds lease tokens/expiry to task and attempt rows, uses them to fence state transitions and heartbeats, and updates staleness detection to rely on lease expiry. It prevents stale or resumed workers from mutating task state after a new attempt has taken over, improving safety under pauses or crashes.