Skip to content

Implement Webhook Retry Engine with Exponential Backoff in Background Worker #282

Description

@Cjay-Cyber-2

Description

The background worker handles asynchronous events such as settlement notifications and external anchor updates. When external endpoints fail or time out during webhook delivery, requests need structured retry logic with exponential backoff rather than immediate failure or raw retries.

Context & Requirements

  • Background tasks run via src/worker/index.ts.
  • Prisma manages database state for webhook logs and task queues.
  • Webhooks must retry up to 5 times using exponential backoff (e.g., 30s, 2m, 10m, 30m, 2h).
  • Failed delivery metadata (HTTP status, error message, attempt count, next retry timestamp) must be tracked in the database.

Acceptance Criteria

  • Add retry schema fields (attemptCount, nextRetryAt, status, lastError) to webhook delivery records.
  • Implement backoff calculation utility calculating delay based on attempt count.
  • Update worker process in src/worker/index.ts to query pending webhooks where nextRetryAt is less than or equal to current time.
  • Mark webhooks as permanently failed after maximum retry limit (5 attempts).
  • Add unit tests verifying retry calculations and worker dispatch state transitions.

Implementation Guidance

  • Modify Prisma schema in prisma/schema.prisma if retry tracking fields are missing.
  • Implement backoff schedule helper function in src/services/webhookService.ts.
  • Update processing loop in src/worker/index.ts to fetch records using Prisma filtered by status and nextRetryAt.

Testing & Validation

  • Run unit tests with npm test verifying backoff timing functions.
  • Run integration tests with npm run test:integration testing webhook failure state updates.

Submission Guidelines

  • Open a PR that includes "Closes #".
  • Ensure assignment before beginning work.
  • Maintain existing code format and linting with npm run lint.

Wave complexity: Medium

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stellar WaveIssues in the Stellar wave programbackendcomplexity: mediumDrips Wave — moderate scope, some designdrips-waveTracked in a Drips Wave programworker

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions