Skip to content

[oracle] Add alerting service integration with PagerDuty or webhook #921

Description

@Otaiki1

Summary

oracle/src/health/alerting.service.ts exists but likely only logs alerts internally. Operators need to receive external alerts (PagerDuty, Slack webhook, or email) when critical conditions occur: circuit OPEN, DLQ threshold exceeded, or key rotation failure.

What to do

  1. Add ALERT_WEBHOOK_URL and ALERT_PAGERDUTY_KEY to the oracle env schema (both optional).
  2. In AlertingService.sendAlert(level, message, context):
    • If ALERT_WEBHOOK_URL is set, POST a JSON payload to the URL (Slack-compatible format).
    • If ALERT_PAGERDUTY_KEY is set, use PagerDuty's Events API v2.
  3. Trigger alerts for: circuit breaker OPEN, DLQ depth > configurable threshold, VRF key unavailable.
  4. Include oracle_id, raffle_id (if applicable), and severity in the alert payload.
  5. Write a unit test mocking the HTTP client to assert alert payloads are correctly formed.

Acceptance criteria

  • Setting ALERT_WEBHOOK_URL causes alerts to POST to that URL when the circuit opens.
  • Alert payload contains enough context to diagnose the issue without SSH access.

Metadata

Metadata

Assignees

Labels

Stellar WaveFull ecosystem refactor initiative

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions