Monitoring and Observability

This document describes the monitoring setup for Credence Backend, including Prometheus metrics instrumentation and Grafana dashboard configuration.

Overview

The monitoring stack consists of:

Prometheus - Metrics collection and storage
Grafana - Visualization and dashboards
Application metrics - Custom business and infrastructure metrics

Health Endpoints

The health router separates liveness and readiness:

GET /api/health/live: process-level liveness only (always 200 while process is up).
GET /api/health and GET /api/health/ready: deep readiness checks for Postgres, Redis, Horizon listener heartbeat, and outbox publisher state.

The outbox readiness check now also evaluates the age of the oldest unpublished outbox event. If that lag exceeds 60 seconds, readiness fails and the dependency is reported as down with a lagSeconds value.

Readiness responses include per-check status (up, down, not_configured) and safe diagnostic fields (for example heartbeat age or outbox lag) without exposing secrets such as connection strings.

Architecture

┌─────────────────┐
│ Credence Backend│
│   (Express)     │──── Exposes /metrics endpoint
└────────┬────────┘
         │
         │ scrapes
         ▼
┌─────────────────┐
│   Prometheus    │──── Stores time-series data
└────────┬────────┘
         │
         │ queries
         ▼
┌─────────────────┐
│    Grafana      │──── Visualizes metrics
└─────────────────┘

Metrics Instrumentation

Required Dependencies

Add Prometheus client library to your project:

npm install prom-client

Metrics Implementation

Create src/middleware/metrics.ts:

import { Request, Response, NextFunction } from 'express'
import client from 'prom-client'

// Create a Registry
export const register = new client.Registry()

// Add default metrics (CPU, memory, etc.)
client.collectDefaultMetrics({ register })

// HTTP Metrics
export const httpRequestsTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status'],
  registers: [register]
})

// Health Check Metrics
export const healthCheckStatus = new client.Gauge({
  name: 'health_check_status',
  help: 'Health check status (1 = up, 0 = down)',
  labelNames: ['dependency'],
  registers: [register]
})

export const healthCheckDuration = new client.Gauge({
  name: 'health_check_duration_seconds',
  help: 'Duration of health checks in seconds',
  labelNames: ['dependency'],
  registers: [register]
})

// Business Metrics
export const reputationScoreCalculations = new client.Counter({
  name: 'reputation_score_calculations_total',
  help: 'Total number of reputation score calculations',
  registers: [register]
})

export const reputationCalculationDuration = new client.Histogram({
  name: 'reputation_calculation_duration_seconds',
  help: 'Duration of reputation calculations in seconds',
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
  registers: [register]
})

export const identityVerifications = new client.Counter({
  name: 'identity_verifications_total',
  help: 'Total number of identity verifications',
  labelNames: ['status'],
  registers: [register]
})

export const bulkVerifications = new client.Counter({
  name: 'bulk_verifications_total',
  help: 'Total number of bulk verification requests',
  labelNames: ['status'],
  registers: [register]
})

export const bulkVerificationBatchSize = new client.Histogram({
  name: 'bulk_verification_batch_size',
  help: 'Size of bulk verification batches',
  buckets: [1, 5, 10, 25, 50, 75, 100],
  registers: [register]
})

export const identitySyncDuration = new client.Histogram({
  name: 'identity_sync_duration_seconds',
  help: 'Duration of identity state sync operations',
  labelNames: ['operation'],
  buckets: [0.1, 0.5, 1, 2, 5, 10, 30],
  registers: [register]
})

// Middleware to track HTTP metrics
export function metricsMiddleware(req: Request, res: Response, next: NextFunction) {
  const start = Date.now()
  
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000
    const route = req.route?.path || req.path
    
    httpRequestsTotal.inc({
      method: req.method,
      route,
      status: res.statusCode
    })
    
    httpRequestDuration.observe({
      method: req.method,
      route,
      status: res.statusCode
    }, duration)
  })
  
  next()
}

Integrate Metrics into Application

Update src/index.ts:

import express from 'express'
import { metricsMiddleware, register } from './middleware/metrics.js'
import { createHealthRouter } from './routes/health.js'
import { createDefaultProbes } from './services/health/probes.js'

const app = express()
const PORT = process.env.PORT ?? 3000

app.use(express.json())

// Add metrics middleware
app.use(metricsMiddleware)

// Metrics endpoint for Prometheus
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType)
  res.end(await register.metrics())
})

const healthProbes = createDefaultProbes()
app.use('/api/health', createHealthRouter(healthProbes))

// ... rest of your routes

app.listen(PORT, () => {
  console.log(`Credence API listening on http://localhost:${PORT}`)
})

export default app

Instrument Health Checks

Update src/services/health/checks.ts to emit metrics:

import { healthCheckStatus, healthCheckDuration } from '../../middleware/metrics.js'

export async function runHealthChecks(probes: {
  db?: HealthProbe
  redis?: HealthProbe
  external?: HealthProbe
}): Promise<HealthResult> {
  // Run checks and measure duration
  const dbStart = Date.now()
  const db = probes.db ? await probes.db() : { status: 'not_configured' as const }
  healthCheckDuration.set({ dependency: 'db' }, (Date.now() - dbStart) / 1000)
  healthCheckStatus.set({ dependency: 'db' }, db.status === 'up' ? 1 : 0)

  const redisStart = Date.now()
  const redis = probes.redis ? await probes.redis() : { status: 'not_configured' as const }
  healthCheckDuration.set({ dependency: 'redis' }, (Date.now() - redisStart) / 1000)
  healthCheckStatus.set({ dependency: 'redis' }, redis.status === 'up' ? 1 : 0)

  // ... rest of health check logic
}

Instrument Business Operations

Update src/services/identityService.ts:

import { identityVerifications, bulkVerifications, bulkVerificationBatchSize } from '../middleware/metrics.js'

export class IdentityService {
  async verifyIdentity(address: string): Promise<IdentityVerification> {
    try {
      // ... verification logic
      identityVerifications.inc({ status: 'success' })
      return result
    } catch (error) {
      identityVerifications.inc({ status: 'error' })
      throw error
    }
  }

  async verifyBulk(addresses: string[]): Promise<{
    results: IdentityVerification[]
    errors: VerificationError[]
  }> {
    bulkVerificationBatchSize.observe(addresses.length)
    
    try {
      // ... bulk verification logic
      bulkVerifications.inc({ status: 'success' })
      return { results, errors }
    } catch (error) {
      bulkVerifications.inc({ status: 'error' })
      throw error
    }
  }
}

Update src/services/reputation/score.ts:

import { reputationScoreCalculations, reputationCalculationDuration } from '../../middleware/metrics.js'

export function calculateReputationScore(input: ReputationInput): ReputationScore {
  const start = Date.now()
  
  // ... calculation logic
  
  reputationScoreCalculations.inc()
  reputationCalculationDuration.observe((Date.now() - start) / 1000)
  
  return result
}

Update src/listeners/identityStateSync.ts:

import { identitySyncDuration } from '../middleware/metrics.js'

export class IdentityStateSync {
  async reconcileByAddress(address: string): Promise<ReconcileResult> {
    const start = Date.now()
    
    try {
      // ... reconciliation logic
      return result
    } finally {
      identitySyncDuration.observe({ operation: 'reconcile' }, (Date.now() - start) / 1000)
    }
  }

  async fullResync(): Promise<FullResyncResult> {
    const start = Date.now()
    
    try {
      // ... full resync logic
      return result
    } finally {
      identitySyncDuration.observe({ operation: 'full_resync' }, (Date.now() - start) / 1000)
    }
  }
}

Prometheus Configuration

Prometheus Setup

Create monitoring/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'credence-production'
    environment: 'production'

scrape_configs:
  - job_name: 'credence-backend'
    static_configs:
      - targets: ['localhost:3000']
    metrics_path: '/metrics'
    scrape_interval: 10s
    scrape_timeout: 5s

Running Prometheus

Using Docker:

docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v $(pwd)/monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Using Docker Compose (create docker-compose.yml):

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3001:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana-data:/var/lib/grafana
      - ./monitoring/grafana:/etc/grafana/provisioning/dashboards
    depends_on:
      - prometheus
    restart: unless-stopped

volumes:
  prometheus-data:
  grafana-data:

Start the stack:

docker-compose up -d

Grafana Dashboard

Dashboard Overview

The Credence Backend dashboard (monitoring/grafana/dashboard.json) provides comprehensive monitoring across:

HTTP Metrics
- Error rate (5xx responses)
- Request rate by endpoint
- Request latency (p50, p95)
- Status code distribution
Infrastructure Health
- Database health status
- Redis health status
- Health check duration
Business Metrics
- Reputation score calculations
- Identity verifications
- Bulk verification operations
- Batch size distribution
- Operation duration (p95)

Importing the Dashboard

Method 1: Grafana UI

Open Grafana at http://localhost:3001 (default credentials: admin/admin)
Navigate to Dashboards → Import
Click Upload JSON file
Select monitoring/grafana/dashboard.json
Select your Prometheus data source
Click Import

Method 2: Provisioning (Automated)

Create monitoring/grafana/provisioning/dashboards/dashboard.yml:

apiVersion: 1

providers:
  - name: 'Credence Dashboards'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards

Create monitoring/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true

Update docker-compose.yml to mount provisioning configs:

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana-data:/var/lib/grafana
      - ./monitoring/grafana/provisioning:/etc/grafana/provisioning
      - ./monitoring/grafana/dashboard.json:/etc/grafana/provisioning/dashboards/credence-backend.json

Restart Grafana:

docker-compose restart grafana

The dashboard will be automatically imported and available.

Dashboard Panels

Panel 1: HTTP Error Rate (5xx)

Type: Gauge
Query: rate(http_requests_total{job="credence-backend", status=~"5.."}[5m]) / rate(http_requests_total{job="credence-backend"}[5m])
Purpose: Monitor server error rate; alerts when > 5%

Panel 2: HTTP Request Rate

Type: Time series
Query: rate(http_requests_total{job="credence-backend"}[5m])
Purpose: Track request volume by endpoint and status

Panel 3: HTTP Request Latency

Type: Time series
Queries:
- p95: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
- p50: histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m]))
Purpose: Monitor response times; identify slow endpoints

Panel 4: HTTP Status Codes Distribution

Type: Time series (stacked)
Query: rate(http_requests_total{job="credence-backend"}[5m])
Purpose: Visualize 2xx, 4xx, 5xx distribution

Panel 5: Database Health

Type: Gauge
Query: health_check_status{job="credence-backend", dependency="db"}
Purpose: Real-time DB connectivity status

Panel 6: Redis Health

Type: Gauge
Query: health_check_status{job="credence-backend", dependency="redis"}
Purpose: Real-time Redis connectivity status

Panel 7: Health Check Duration

Type: Time series
Queries:
- DB: health_check_duration_seconds{dependency="db"}
- Redis: health_check_duration_seconds{dependency="redis"}
Purpose: Monitor health check performance

Panel 8: Business Metrics - Operations Rate

Type: Time series
Queries:
- rate(reputation_score_calculations_total[5m])
- rate(identity_verifications_total[5m])
- rate(bulk_verifications_total[5m])
Purpose: Track business operation volume

Panel 9: Business Operations Duration (p95)

Type: Time series
Queries:
- histogram_quantile(0.95, rate(reputation_calculation_duration_seconds_bucket[5m]))
- histogram_quantile(0.95, rate(identity_sync_duration_seconds_bucket[5m]))
Purpose: Monitor performance of critical business operations

Panel 10: Avg Bulk Verification Batch Size

Type: Gauge
Query: avg(bulk_verification_batch_size)
Purpose: Track average batch size for capacity planning

Panel 11: Total Verifications (24h)

Type: Stat
Query: sum(increase(identity_verifications_total[24h]))
Purpose: Daily verification volume

Alerting

Recommended Alerts

Create monitoring/prometheus/alerts.yml:

groups:
  - name: credence_backend_alerts
    interval: 30s
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{job="credence-backend", status=~"5.."}[5m]) 
          / rate(http_requests_total{job="credence-backend"}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High HTTP error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"

      # High latency
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, 
            rate(http_request_duration_seconds_bucket{job="credence-backend"}[5m])
          ) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High request latency detected"
          description: "P95 latency is {{ $value }}s (threshold: 2s)"

      # Database down
      - alert: DatabaseDown
        expr: health_check_status{job="credence-backend", dependency="db"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Database is down"
          description: "PostgreSQL health check failing"

      # Redis down
      - alert: RedisDown
        expr: health_check_status{job="credence-backend", dependency="redis"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Redis is down"
          description: "Redis health check failing"

      # Slow health checks
      - alert: SlowHealthCheck
        expr: health_check_duration_seconds{job="credence-backend"} > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Health check is slow"
          description: "{{ $labels.dependency }} health check taking {{ $value }}s"

      # Low verification rate (business metric)
      - alert: LowVerificationRate
        expr: rate(identity_verifications_total{job="credence-backend"}[10m]) < 0.1
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Low identity verification rate"
          description: "Verification rate dropped to {{ $value }} req/s"

Update prometheus.yml to include alerts:

rule_files:
  - 'alerts.yml'

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'alertmanager:9093'

Deployment

Kubernetes Deployment

ServiceMonitor for Prometheus Operator

Create k8s/servicemonitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: credence-backend
  namespace: default
  labels:
    app: credence-backend
spec:
  selector:
    matchLabels:
      app: credence-backend
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

ConfigMap for Grafana Dashboard

Create k8s/grafana-dashboard-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: credence-backend-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  credence-backend.json: |
    # Paste contents of monitoring/grafana/dashboard.json here

Apply:

kubectl apply -f k8s/servicemonitor.yaml
kubectl apply -f k8s/grafana-dashboard-configmap.yaml

Production Considerations

Metrics Retention: Configure Prometheus retention based on storage capacity

command:
  - '--storage.tsdb.retention.time=30d'
  - '--storage.tsdb.retention.size=50GB'

High Availability: Deploy Prometheus with replication
```
replicas: 2
```
Remote Storage: Use long-term storage (Thanos, Cortex, or cloud providers)
Security:
- Enable authentication on Grafana
- Restrict Prometheus access
- Use TLS for metrics endpoints in production

Resource Limits: Set appropriate limits

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "1Gi"
    cpu: "500m"

Metrics Endpoint Security

The /metrics endpoint is unauthenticated by design (Prometheus does not send auth headers) but can be restricted to cluster-internal IPs via the METRICS_ALLOWED_CIDRS environment variable.

METRICS_ALLOWED_CIDRS=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

When set, only requests originating from IPs within the listed CIDR ranges can reach /metrics. All other IPs receive 403 Forbidden.

When unset (default), /metrics is open — suitable for local development.

For Kubernetes deployments, set this to the cluster pod CIDR in k8s/configmap.yaml:

data:
  METRICS_ALLOWED_CIDRS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

Testing

Verify Metrics Endpoint

# Check metrics are exposed
curl http://localhost:3000/metrics

# Expected output includes:
# - http_requests_total
# - http_request_duration_seconds
# - health_check_status
# - reputation_score_calculations_total
# - identity_verifications_total

Generate Test Traffic

# Generate some requests
for i in {1..100}; do
  curl http://localhost:3000/api/health
  curl http://localhost:3000/api/trust/GABC123...
done

# Bulk verification test
curl -X POST http://localhost:3000/api/bulk/verify \
  -H "Content-Type: application/json" \
  -H "X-API-Key: test-enterprise-key-12345" \
  -d '{"addresses": ["GABC...", "GDEF..."]}'

Query Prometheus

# Open Prometheus UI
open http://localhost:9090

# Example queries:
# - rate(http_requests_total[5m])
# - histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# - health_check_status

View Dashboard

# Open Grafana
open http://localhost:3001

# Login: admin/admin
# Navigate to: Dashboards → Credence Backend - API Monitoring

Metrics Reference

HTTP Metrics

Metric	Type	Labels	Description
`http_requests_total`	Counter	method, route, status	Total HTTP requests
`http_request_duration_seconds`	Histogram	method, route, status	Request duration

Health Metrics

Metric	Type	Labels	Description
`health_check_status`	Gauge	dependency	Health status (1=up, 0=down)
`health_check_duration_seconds`	Gauge	dependency	Health check duration

Pool Metrics

Metric	Type	Labels	Description
`pg_pool_total_count`	Gauge	pool	Total clients (active + idle)
`pg_pool_idle_count`	Gauge	pool	Idle clients
`pg_pool_waiting_count`	Gauge	pool	Queued requests waiting

Business Metrics

Metric	Type	Labels	Description
`reputation_score_calculations_total`	Counter	-	Total reputation calculations
`reputation_calculation_duration_seconds`	Histogram	-	Calculation duration
`identity_verifications_total`	Counter	status	Total identity verifications
`bulk_verifications_total`	Counter	status	Total bulk verification requests
`bulk_verification_batch_size`	Histogram	-	Batch size distribution
`rate_limit_hits_total`	Counter	tenant, tier	Total rate limit hits grouped by tenant and tier
`identity_sync_duration_seconds`	Histogram	operation	Identity sync duration
`queue_backlog_size`	Gauge	topic	Current number of items pending in the backlog queue per topic; sampled every 15 s

Memory & OOM Metrics

Metric	Type	Labels	Description
`oom_events_total`	Counter	-	Total number of out-of-memory events detected

Memory Configuration

To set a maximum memory limit for Node.js (preventing OOM crashes), use the NODE_MAX_OLD_SPACE_SIZE_MB environment variable. This sets the --max-old-space-size Node.js flag automatically via the entrypoint script.

Example Usage

Local Development

# Set 2GB limit
NODE_MAX_OLD_SPACE_SIZE_MB=2048 npm start

Docker

docker run -e NODE_MAX_OLD_SPACE_SIZE_MB=2048 credence-backend

Docker Compose

services:
  credence-backend:
    image: credence-backend
    environment:
      - NODE_MAX_OLD_SPACE_SIZE_MB=2048

Kubernetes

spec:
  containers:
  - name: credence-backend
    env:
    - name: NODE_MAX_OLD_SPACE_SIZE_MB
      value: "2048"
    resources:
      requests:
        memory: "2Gi"
      limits:
        memory: "2Gi"

The NODE_MAX_OLD_SPACE_SIZE_MB should match your container's memory limit.

Default Metrics (from prom-client)

process_cpu_user_seconds_total - User CPU time
process_cpu_system_seconds_total - System CPU time
process_resident_memory_bytes - Resident memory
nodejs_heap_size_total_bytes - Heap size
nodejs_heap_size_used_bytes - Used heap
nodejs_eventloop_lag_seconds - Event loop lag

Troubleshooting

Metrics Not Appearing

Check metrics endpoint:
```
curl http://localhost:3000/metrics
```
If empty or error, verify prom-client is installed and middleware is registered.
Check Prometheus targets:
- Open http://localhost:9090/targets
- Verify credence-backend target is UP
- If DOWN, check network connectivity and firewall rules
Check Prometheus logs:
```
docker logs prometheus
```

Dashboard Not Loading

Verify data source:
- Grafana → Configuration → Data Sources
- Test Prometheus connection
- Ensure URL is correct (e.g., http://prometheus:9090)
Check dashboard queries:
- Edit panel → Query inspector
- Verify metrics exist in Prometheus
- Check label selectors match your job name
Verify time range:
- Ensure dashboard time range has data
- Try "Last 5 minutes" for recent data

High Cardinality Issues

If metrics storage grows too large:

Limit label values:
- Avoid user IDs or addresses as labels
- Use fixed label sets (e.g., status: success/error)
Adjust retention:
```
--storage.tsdb.retention.time=15d
```

Use recording rules for expensive queries:

groups:
  - name: credence_recordings
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: rate(http_requests_total{job="credence-backend"}[5m])

Performance Impact

Monitor metrics collection overhead:

# Check /metrics response time
time curl http://localhost:3000/metrics

# Should be < 100ms

If slow:

Reduce histogram buckets
Disable default metrics if not needed
Use summary instead of histogram for high-cardinality data

Screenshots

Dashboard Overview

The dashboard provides a comprehensive view of:

Real-time error rates and request volumes
Latency percentiles (p50, p95) across all endpoints
Infrastructure health (DB, Redis) with status indicators
Business metrics showing verification rates and batch sizes

Key Visualizations

Top Row: Error rate gauge, request rate time series, latency percentiles
Middle Row: Status code distribution, DB health, Redis health
Bottom Rows: Health check durations, business operation rates, and daily totals

Next Steps

Install dependencies:
```
npm install prom-client
```
Implement metrics middleware (see code examples above)
Deploy monitoring stack:
```
docker-compose up -d
```
Import dashboard into Grafana
Configure alerts based on your SLOs
Set up notification channels (Slack, PagerDuty, email)

Resources

Support

For issues or questions:

Check the troubleshooting section
Review Prometheus and Grafana logs
Consult the metrics reference for available metrics
Verify network connectivity between services

Uh oh!

FilesExpand file tree

monitoring.md

Latest commit

History

monitoring.md

File metadata and controls

Monitoring and Observability

Overview

Health Endpoints

Architecture

Metrics Instrumentation

Required Dependencies

Metrics Implementation

Integrate Metrics into Application

Instrument Health Checks

Instrument Business Operations

Prometheus Configuration

Prometheus Setup

Running Prometheus

Grafana Dashboard

Dashboard Overview

Importing the Dashboard

Method 1: Grafana UI

Method 2: Provisioning (Automated)

Dashboard Panels

Panel 1: HTTP Error Rate (5xx)

Panel 2: HTTP Request Rate

Panel 3: HTTP Request Latency

Panel 4: HTTP Status Codes Distribution

Panel 5: Database Health

Panel 6: Redis Health

Panel 7: Health Check Duration

Panel 8: Business Metrics - Operations Rate

Panel 9: Business Operations Duration (p95)

Panel 10: Avg Bulk Verification Batch Size

Panel 11: Total Verifications (24h)

Alerting

Recommended Alerts

Deployment

Kubernetes Deployment

ServiceMonitor for Prometheus Operator

ConfigMap for Grafana Dashboard

Production Considerations

Metrics Endpoint Security

Testing

Verify Metrics Endpoint

Generate Test Traffic

Query Prometheus

View Dashboard

Metrics Reference

HTTP Metrics

Health Metrics

Pool Metrics

Business Metrics

Memory & OOM Metrics

Memory Configuration

Example Usage

Local Development

Docker

Docker Compose

Kubernetes

Default Metrics (from prom-client)

Troubleshooting

Metrics Not Appearing

Dashboard Not Loading

High Cardinality Issues

Performance Impact

Screenshots

Dashboard Overview

Key Visualizations

Next Steps

Resources

Support