Webhook delivery retry: retry failed webhook calls with exponential backoff

[leocagli]

Context

The OS event bus fires webhooks on events like agent.status, quest.completed, payment.received. Currently if the downstream endpoint is down, the call is dropped. This issue adds retry logic.

What to implement

Retry store

Track pending deliveries with:

interface PendingDelivery {
  id: string
  webhookUrl: string
  payload: object
  attempts: number       // max 4
  nextRetryAt: number    // epoch ms
}

Retry schedule (exponential backoff)

• Attempt 1: immediate
• Attempt 2: +30s
• Attempt 3: +2m
• Attempt 4: +10m
• After 4 failures: mark as dead_letter, stop retrying

Route: GET /api/webhooks/dead-letters

Returns all dead-lettered deliveries (last 100) for manual inspection.

Route: POST /api/webhooks/dead-letters/:id/retry

Manually re-enqueue a dead-lettered delivery.

Acceptance criteria

• Successful delivery on first attempt: no retry entry created
• Failed delivery retried 3 more times with correct backoff delays
• After 4 failures: delivery appears in GET /api/webhooks/dead-letters
• Manual retry via POST clears dead-letter status and re-attempts
• Unit tests cover: success, partial retry, full dead-letter, manual retry

[aliyuHabibu]

Hi, Please assign the issue to me