[Enhancement] Signed webhook event system: HMAC-signed payment lifecycle callbacks with retries and dead-letter redelivery

[zeemscript]

Summary

Build an outbound webhook/event system so external consumers — the separate DeenBridge AI service, educator tooling, analytics — can subscribe to payment and enrollment lifecycle events (payment.confirmed, payment.failed, course.enrolled, ...) over signed HTTP callbacks. Deliveries carry an HMAC-SHA256 signature and timestamp (Stripe-style), are retried with exponential backoff, and land in a dead-letter state with a redelivery API when a consumer stays down. Today the platform emits nothing: every payment state change is visible only by polling the REST API.

Current state

• All payment lifecycle transitions happen inline in src/controllers/stellar/paymentController.js — initializePayment creates a pending Transaction, submitPayment moves it to submitted then confirmed (or failed with failureReason), and cancelTransaction sets expired. None of these emit any event beyond a logger.info/logger.error line.
• Free-course enrollment is a second, separate write path: enrollInCourse in src/controllers/courses/courseController.js updates Course.enrolledUsers and User.purchasedCourses with no event either.
• There is no outbound-callback machinery of any kind: no webhook model, no delivery worker, no signing. axios is already a dependency (package.json) and Node's crypto covers HMAC — no new packages are strictly required.
• The existing notification system (src/controllers/notificationController.js, src/models/Notification.js) is inbound/in-app (user-facing, SSE) and is being fixed under issue [Enhancement] Wire up and harden the notification system (dead producers, N+1 fan-out, SSE scaling) #22 — it is not a machine-to-machine integration surface and has no signatures, retries, or subscriptions.
• Transaction (src/models/Transaction.js) has the fields consumers need in a payload (buyer/creator, itemType, itemId, amount, stellarTxHash, status, network), so event payloads can be assembled without new queries.

What to build

1. Models:
   • src/models/WebhookEndpoint.js: url (https-only in production), secret (generated server-side, returned once, stored hashed or encrypted — document the choice), events (subscribed event-type list, ["*"] allowed), isActive, description, failure counters, disabledAt/disabledReason (auto-disable after sustained failure).
   • src/models/WebhookDelivery.js: endpoint ref, eventId, eventType, payload snapshot, attempts (array of { at, statusCode, error, durationMs }), status (pending, delivered, retrying, dead), nextAttemptAt (indexed). Cap stored attempt/response bodies to a bounded size.
2. Event catalog + emitter (src/services/webhooks/webhookService.js): a typed emitEvent(type, data) with an initial catalog: payment.initialized, payment.confirmed, payment.failed, payment.expired, course.enrolled, wallet.connected, wallet.disconnected. Each event gets a unique eventId (used for consumer-side idempotency), type, createdAt, apiVersion, and a data object that never includes secrets or full user documents (explicit field allowlist: ids, wallet public keys, amounts, tx hash, item refs). Emitting must be fire-and-forget for the request path: persist WebhookDelivery rows for matching endpoints and return — never block or fail the HTTP response on delivery (emit after the Mongo session commits, not inside it).
3. Signing: headers X-DeenBridge-Event, X-DeenBridge-Event-Id, X-DeenBridge-Timestamp, and X-DeenBridge-Signature: v1=<hex hmac-sha256(secret, timestamp + "." + rawBody)>. Document consumer verification (constant-time compare, reject stale timestamps > 5 min) in the README, including a copy-paste Node verifier snippet.
4. Delivery worker (src/services/webhooks/deliveryWorker.js): interval loop claiming due deliveries (status: pending|retrying, nextAttemptAt <= now) with an atomic findOneAndUpdate claim so multiple instances don't double-send; POST with a strict timeout (~10s) and no redirect following; 2xx → delivered, anything else → schedule retry with exponential backoff + jitter (e.g. 1m, 5m, 30m, 2h, 12h), then dead after max attempts. Auto-disable an endpoint after N consecutive dead deliveries and emit a log warning. Design the worker so it can later be swapped onto the job queue from issue [Enhancement] Background job queue for transactional email, receipts, and Horizon verification retries #32 without schema changes (keep scheduling state in the delivery document, not in memory).
5. Management API (src/routes/webhookRoutes.js, mounted at /api/webhooks in app.js, behind protect + a privilege gate consistent with issue [Enhancement] Introduce role-based authorization and fix registration privilege escalation #20): CRUD for endpoints (secret shown only at creation; a rotate-secret action), GET /api/webhooks/:id/deliveries (paginated, filterable by status), POST /api/webhooks/:id/deliveries/:deliveryId/redeliver (dead → pending), and POST /api/webhooks/:id/ping (sends a signed ping event for integration testing).
6. Wire the emitters: submitPayment (confirmed + failed branches), cancelTransaction, initializePayment, enrollInCourse, connectWallet/disconnectWallet (src/controllers/stellar/walletController.js). Keep each call one line: emitEvent("payment.confirmed", {...}).
7. SSRF guard: validate endpoint URLs at registration and delivery time — https required outside development, and reject loopback/RFC-1918/link-local targets after DNS resolution in production. Document the limitation.

Acceptance criteria

• Confirming a payment produces a delivery row per subscribed endpoint and a signed POST whose HMAC verifies against the documented scheme (test with a local receiver using the README snippet).
• A consumer returning 500 is retried on the backoff schedule (fake timers in tests), transitions to dead after max attempts, and can be successfully redelivered via the API.
• Delivery is fully decoupled from the request path: with an unreachable endpoint registered, POST /api/stellar/payment/submit latency and status are unaffected, and events are emitted only after the DB transaction commits (a rolled-back payment emits nothing).
• Signature scheme rejects tampered bodies and stale timestamps; secrets are never returned by read endpoints after creation; payloads contain only allowlisted fields (no password hashes, emails, or full user docs).
• Endpoint auto-disables after sustained failures and can be re-enabled; ping endpoint delivers a verifiable signed event.
• Two concurrent worker loops never double-deliver the same delivery row (claim is atomic — covered by a test).
• Registration rejects non-https (production) and private-network URLs; management API rejects unauthenticated/unprivileged callers.
• Jest coverage for: HMAC generation/verification, backoff schedule, claim atomicity, payload allowlist, and emitter wiring in submitPayment (mocked axios; no live network).

Pointers

• src/controllers/stellar/paymentController.js (all payment state transitions to instrument; note the Mongo session — emit after commit), src/controllers/courses/courseController.js (enrollInCourse), src/controllers/stellar/walletController.js, src/models/Transaction.js (payload source fields), app.js (route mounting), src/middlewares/authMiddleware.js.
• Related issues to align with, not duplicate: [Enhancement] Wire up and harden the notification system (dead producers, N+1 fan-out, SSE scaling) #22 (in-app notifications — different surface), [Enhancement] Background job queue for transactional email, receipts, and Horizon verification retries #32 (job queue — design for a later swap, don't depend on it), [Enhancement] Introduce role-based authorization and fix registration privilege escalation #20 (role gate for the management API), [Enhancement] Observability: structured pino logging with request IDs, Prometheus /metrics, and slow-query logging #33 (metrics counters live there).
• Prior art worth mirroring: Stripe webhook signatures (https://docs.stripe.com/webhooks) and Svix's retry-schedule conventions.
• Gotchas: compute the HMAC over the exact serialized bytes you send (serialize once, sign, POST the same buffer) — re-serializing JSON can reorder keys and break verification. Use crypto.timingSafeEqual in the documented verifier. CI (.github/workflows/ci.yml) has no outbound network, so all delivery tests must mock axios; Jest runs via node --experimental-vm-modules node_modules/jest/bin/jest.js. PRs target dev.

Difficulty

Medium — well-trodden design space, but doing it credibly requires transaction-commit-safe emission, atomic claim semantics, correct HMAC handling, SSRF guards, and a full retry/dead-letter lifecycle with tests.

---

🏆 GrantFox OSS — Official Campaign | FWC26. Apply for this issue through the GrantFox campaign page. The maintainer assigns one contributor before work starts; unassigned PRs may not be reviewed. PRs target the dev branch. Quality bar: CI must stay green.

💬 Questions or need help? Reach the maintainers and other contributors on the DeenBridge Telegram: https://t.me/+nst9lXNj1wc4ZDE0

[Johnsource-hub]

Hello Maintainer,

I'd like to work on this issue. I've reviewed the requirements and can complete the implementation as specified.

Please assign it to me. Thank you!

[code-cannot-be-blank]

Hi Maintainer, please permit me to take on this

[grantfox-oss]

🦊 GrantFox — @Johnsource-hub has been assigned to this issue as part of the Official Campaign | FWC26 campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #45)
2. Your PR will be reviewed by the Deen-Bridge maintainers

Good luck! Track your progress on GrantFox.