Add TW retry queue and wire it into webhooks and escrows

[AnoukRImola]

Trustless Work Retry & Recovery Queue

Closes #63

Implements the foundational retry/recovery queue for Trustless Work (TW) operations and
connects it to the two places in the backend that talk to TW today (inbound webhooks and
outbound escrow writes), so network failures and transient TW errors no longer permanently
fail an Agreement.

Why

Network failures, temporary TW outages, or blockchain transaction delays must not permanently
fail Agreement execution. Before this PR, src/webhooks/webhooks.service.ts had its own
in-memory retry loop (withRetry), and the outbound TW calls in
src/internal-trustless/escrow-write.helper.ts had no retry at all — a single failure meant a
lost operation. This PR builds the one shared retry primitive the issue calls for, so every
module reuses it instead of accumulating competing retry implementations.

What's included

1. The retry queue itself (src/retry-queue)

• RetryQueueService.enqueue(jobType, payload, idempotencyKey, options?) — persists a job to
  the retry_jobs table (Postgres/Supabase). A duplicate idempotencyKey is a no-op that
  returns the existing job (including a race-safe path for concurrent inserts).
• RetryQueueService.registerHandler(jobType, handler) — one handler per RetryJobType,
  typically registered in a consumer module's onModuleInit. The handler must throw to signal
  failure; the queue treats any rejection as retryable.
• A poller (setInterval, default every 5s) picks up due pending jobs and processes them with
  exponential backoff (base * 2^(attempt-1), capped) until max_attempts is reached, then
  marks the job failed.
• Crash recovery: jobs stuck in processing past a staleness threshold are reclaimed back to
  pending on the next poll — nothing is lost on a process restart.
• Admin-only manual retry: POST /v1/retry-queue/:id/retry re-runs a job once, bypassing its
  scheduled backoff. GET /v1/retry-queue / GET /v1/retry-queue/:id for inspection, both
  gated on profiles.role === 'admin'.
• Fully configurable via env: RETRY_QUEUE_MAX_ATTEMPTS, RETRY_QUEUE_BASE_DELAY_MS,
  RETRY_QUEUE_MAX_DELAY_MS, RETRY_QUEUE_POLL_INTERVAL_MS, RETRY_QUEUE_CONCURRENCY,
  RETRY_QUEUE_STALE_PROCESSING_MS — see .env.example.
• RetryQueueModule is @Global(), registered in AppModule — any module can inject
  RetryQueueService without importing the module.
• DB migration: scripts/002_create_retry_jobs.sql (table + indexes + RLS).

2. Webhooks now reuse the queue instead of retrying inline

• WebhooksService.withRetry() (in-memory, no persistence, competing retry logic) is removed.
• handleEvent() now enqueues a WEBHOOK_EVENT_PROCESSING job and ACKs 200 immediately,
  instead of holding the HTTP connection open for up to ~7s of inline retries.
• The actual DB writes (applyStatusUpdate / applyMilestoneUpdate / applyInfoUpdate,
  including the existing idempotent-duplicate guard) now run as the registered handler, driven
  by the queue's poller with real backoff and persistence.
• Idempotency key is derived from contractId:event:hash(payload), since TW webhook payloads
  carry no event id.
• New job type + migration: scripts/003_add_webhook_retry_job_type.sql adds
  webhook_event_processing to the retry_jobs.job_type check constraint.

3. Escrow writes get a retry backstop (without changing the frontend contract)

src/internal-trustless/escrows.controller.ts write endpoints (create, fund,
approve-milestone, change-milestone-status, release) still call Trustless Work
synchronously and return the result (e.g. the unsigned XDR) exactly as before — the frontend
signing flow is unaffected on the happy path.

On a transient failure (TW 5xx or a network-level error), the request still fails the same way
it did before, but the operation is now also enqueued on the shared queue
(AGREEMENT_CREATION / MILESTONE_UPDATE / PAYMENT_EXECUTION, keyed by a deterministic
idempotency key per operation) so it isn't silently dropped and can recover in the background or
be manually retried by an admin. Validation errors (4xx) are not retried — retrying a bad
request can't help.

Scope boundaries (intentionally left out)

• CONTRACT_RETRIEVAL (getEscrowsBySigner / getEscrowsByRole) — read-only GETs with no
  side effect to recover; the caller has already moved on by the time a background retry would
  finish, so wiring these to the queue wouldn't do anything useful.
• STATUS_SYNC — no producer exists yet. There is no periodic reconciliation job in this
  backend (no cron/scheduler); building that is the job of the dependent "Agreement
  Synchronization" issue, not this one. The job type is defined and ready to use once that
  producer exists.
• disputeMilestone / sendTransaction — don't map to any of the 5 operation types the
  issue defines; left untouched.
• AgreementsService — pure Supabase CRUD, makes no Trustless Work calls, so it's out of
  scope for a Trustless Work retry primitive.

Testing

• Unit: backoff schedule, max-attempts cutoff, idempotency/duplicate prevention
  (src/retry-queue/retry-queue.spec.ts).
• Integration: a mocked failing TW call retried and recovered end-to-end, duplicate
  idempotency-key blocking, admin manual retry via HTTP (incl. 403 for non-admins), and a new
  case proving WebhooksService reuses the shared queue end-to-end — enqueue → poller → DB
  update (src/integration/retry-queue.integration.spec.ts).
• Restart/persistence: a job left processing by a simulated crash is reclaimed and resumes.
• src/webhooks/webhooks.spec.ts — rewritten to verify enqueue-and-ACK behavior and that the
  registered handler still runs the existing business logic.
• src/internal-trustless/escrows.controller.spec.ts — new: backstop enqueues on 5xx/network
  errors, does not enqueue on 4xx, happy path is unchanged, generic replay handler works.

All 102 tests pass across 9 suites; tsc --noEmit and eslint are clean.

Files changed

.env.example                                        | retry-queue env vars documented
README.md                                            | retry-queue module + endpoints documented
docs/integration-tests.md                            | retry-queue integration suite documented
package.json / pnpm-lock.yaml                        | test:integration script, jest-environment-node
scripts/002_create_retry_jobs.sql                    | retry_jobs table (new)
scripts/003_add_webhook_retry_job_type.sql            | adds webhook_event_processing job type (new)
src/app.module.ts                                     | registers RetryQueueModule
src/retry-queue/*                                     | the retry queue itself (new)
src/webhooks/webhooks.service.ts                      | reuses the queue instead of its own retry loop
src/webhooks/webhooks.spec.ts                         | updated for the new flow
src/internal-trustless/escrow-write.helper.ts         | exports relayWrite + request builders, TrustlessRelayError
src/internal-trustless/escrows.controller.ts          | backstop enqueue on transient TW failures
src/internal-trustless/escrows.controller.spec.ts     | new
src/integration/retry-queue.integration.spec.ts       | end-to-end retry queue + webhook integration coverage
src/integration/migrated-flows.integration.spec.ts    | provides RetryQueueService for EscrowsController's DI

[vercel]

@AnoukRImola is attempting to deploy a commit to the ManuelJG's projects Team on Vercel.

A member of the Team first needs to authorize it.

[Kalchaqui]

Review — almost LGTM (one required fix)

Strong PR for #63. Scope, API surface, and wiring look right.

What looks good

• RetryQueueService.enqueue(jobType, payload, idempotencyKey) with DB persistence + unique-key race handling
• Exponential backoff + max attempts + stale processing reclaim (restart recovery)
• Admin list/get/POST :id/retry gated on profiles.role === 'admin'
• Env knobs documented in .env.example
• Webhooks drop the in-memory withRetry() and reuse this queue (enqueue + ACK)
• Escrow writes: sync happy path preserved; 5xx/network backstop enqueue; 4xx not retried
• Unit + integration coverage is meaningful — local retry-queue suites: 20/20 passed

Intentionally deferred STATUS_SYNC / CONTRACT_RETRIEVAL producers until the sync engine (#60) is fine for a foundational issue.

Required before merge

Rename SQL migrations — main already has scripts/002_create_kyb_verifications.sql. This PR adds another 002_* (+ 003_*). Please renumber to the next free sequence, e.g.:

• scripts/003_create_retry_jobs.sql
• scripts/004_add_webhook_retry_job_type.sql

(or fold the webhook job_type into a single 003 migration). Two different 002_* files will confuse anyone applying scripts in order.

Non-blocking notes

1. Escrow create backstop retries the same {path,body} after a lost response — if TW already accepted the deploy, a blind replay can double-create unless TW is idempotent on engagementId. Worth a short comment / follow-up when Implement Trustless Work Agreement Synchronization Engine #60 lands.
2. Poller setInterval keeps the process handle open (Jest needs --forceExit) — fine for Nest, just watch CI timeouts.

After the migration rename + green CI, this is good to merge from my side. Nice work @AnoukRImola.