Skip to content

RUNBOOKS.md

Latest commit

 

History

History
43 lines (34 loc) · 2.57 KB

RUNBOOKS.md

File metadata and controls

43 lines (34 loc) · 2.57 KB
Raw

Runbooks for Critical Alerts

This document provides actionable steps for the most common critical alerts generated by the teachLink_backend alerting service.

1. PAYMENT_FAILURE_RATE_CRITICAL

Description: The payment failure rate has exceeded the critical threshold (default > 5%). Potential Impact: Users cannot successfully complete checkouts, resulting in immediate revenue loss.

Mitigation Steps:

  1. Check the Stripe dashboard (or equivalent payment provider) for ongoing global incidents.
  2. Review the backend application logs for PaymentsModule errors. Look for API key expiration or network timeouts.
  3. If the payment gateway API is down, update the Statuspage to inform users of the degradation.
  4. Verify webhooks are successfully processing by checking the stripe-webhook.guard.ts logs.

2. ACTIVE_CONNECTIONS_CRITICAL

Description: Database connection pool usage has exceeded 95%. Potential Impact: The application cannot serve requests that require database access, leading to cascading HTTP 5xx errors.

Mitigation Steps:

  1. Identify any long-running transactions or deadlocks using pg_stat_activity.
  2. Check if a recent deployment introduced N+1 queries.
  3. If necessary, temporarily scale up the database connection pool in the environment variables and restart the pods.
  4. If legitimate traffic spikes caused the exhaustion, evaluate horizontally scaling the database read-replicas.

3. MEMORY_USAGE_CRITICAL

Description: Application memory usage has exceeded 95%. Potential Impact: OOM (Out of Memory) crashes resulting in pod restarts and degraded performance.

Mitigation Steps:

  1. Examine the application performance metrics for memory leaks (e.g., unbounded array growth, unresolved Promises).
  2. Look for large file processing jobs in the MediaModule that might be loading full files into memory rather than streaming.
  3. If a specific pod is consistently running out of memory, manually restart it.
  4. Consider scaling up the pod memory limits if the baseline usage has naturally increased.

4. HTTP_ERROR_RATE_CRITICAL

Description: HTTP 5xx error rate has exceeded 5%. Potential Impact: Users are experiencing a degraded application experience.

Mitigation Steps:

  1. Check Datadog/Kibana for the most frequent 5xx errors and their stack traces.
  2. Correlate the spike with recent deployments or infrastructure changes.
  3. Roll back the latest deployment if a specific PR is identified as the root cause.
  4. If the errors correlate with a specific downstream microservice outage, ensure Circuit Breakers are correctly failing fast.