Issue: Implement Trustless Work Retry & Recovery Queue
Depends on: — (foundational — must be built before the sync/webhook issues)
Task Description
Implement a single, shared retry and recovery mechanism for failed Trustless Work (TW) operations. Network failures, temporary service outages, or blockchain transaction delays must not permanently fail Agreement execution. This queue is the one retry primitive the whole backend uses — the synchronization, webhook, milestone, and lifecycle features must reuse it instead of implementing their own retry logic.
Deliverable
A resilient, persistent retry queue capable of recovering failed Trustless Work requests, exposed as a reusable service that other modules call.
Requirements
- Support retries for these operation types:
- Agreement creation
- Milestone updates
- Status synchronization
- Contract retrieval
- Payment execution
- Prevent duplicate executions (idempotency keys per job).
- Log every retry attempt (attempt number, error, outcome).
- Configurable maximum retry attempts.
- Expose a clear public API (e.g.
enqueue(jobType, payload, idempotencyKey)) so other services never re-implement retries.
Validation System
- Simulate a failing TW call and confirm the job is retried with backoff and eventually succeeds or is marked failed after max attempts.
- Enqueue the same job twice with the same idempotency key and confirm it executes only once.
- Confirm a persisted job survives a process restart and resumes.
Technical Requirements
- Exponential backoff (configurable base/max).
- Configurable via env/config (max attempts, backoff, concurrency).
- Persist failed jobs (DB-backed table or queue store) so nothing is lost on restart.
- Allow manual retries (admin-triggered re-run of a failed job).
- Generate meaningful, structured logs for every attempt.
Additional Notes
This feature is critical for production, where external API failures must not break Agreement consistency. This must be sequenced first. The webhook, Agreement Synchronization, milestone, and lifecycle issues must be edited to say "reuse the retry queue from this issue" rather than specifying their own retry mechanism — otherwise the backend ends up with 4–5 competing retry implementations.
Testing (required)
Proof of Completion (required)
Issue: Implement Trustless Work Retry & Recovery Queue
Depends on: — (foundational — must be built before the sync/webhook issues)
Task Description
Implement a single, shared retry and recovery mechanism for failed Trustless Work (TW) operations. Network failures, temporary service outages, or blockchain transaction delays must not permanently fail Agreement execution. This queue is the one retry primitive the whole backend uses — the synchronization, webhook, milestone, and lifecycle features must reuse it instead of implementing their own retry logic.
Deliverable
A resilient, persistent retry queue capable of recovering failed Trustless Work requests, exposed as a reusable service that other modules call.
Requirements
enqueue(jobType, payload, idempotencyKey)) so other services never re-implement retries.Validation System
Technical Requirements
Additional Notes
This feature is critical for production, where external API failures must not break Agreement consistency. This must be sequenced first. The webhook, Agreement Synchronization, milestone, and lifecycle issues must be edited to say "reuse the retry queue from this issue" rather than specifying their own retry mechanism — otherwise the backend ends up with 4–5 competing retry implementations.
Testing (required)
Proof of Completion (required)