fix: make Prometheus alert rules runtime-agnostic and add comprehensive production alerts

[DivyashreeR008]

Summary

Fixes #60 by replacing Go-only Prometheus metrics (http_requests_total, process_resident_memory_bytes) with runtime-agnostic alternatives and expanding from 3 to 12 production-grade alert rules.

Changes

Bugs Fixed

• HighErrorRate: Replaced http_requests_total{status=~"5.."} (Go-only, doesn't exist in Node.js/Python) with atlas_http_requests_total{status_code=~"5[0-9][0-9]"} — the actual metric name exposed by Python services via the atlas_observability library. Also changed from raw error rate to error percentage (errors/total requests).
• HighMemoryUsage: Removed process_resident_memory_bytes / process_virtual_memory_bytes (Go runtime metrics, not available in Node.js, Python, or Java). Replaced with node-level memory alerts using node_memory_* metrics.

New Alerts Added

Alert	Metric Source	Severity	Description
HighLatencyP99	atlas_http_request_duration_seconds	warning	p99 latency > 1s for 5m
HighLatencyP99Critical	atlas_http_request_duration_seconds	critical	p99 latency > 2.5s for 5m
HighInProgressRequests	atlas_http_requests_in_progress	warning	> 100 concurrent requests
DiskSpaceWarning	node_filesystem_*	warning	< 10% free for 5m
DiskSpaceCritical	node_filesystem_*	critical	< 5% free for 1m
NodeMemoryUsageWarning	node_memory_*	warning	> 90% used for 5m
NodeMemoryUsageCritical	node_memory_*	critical	> 95% used for 2m
NodeCPUUsageHigh	node_cpu_seconds_total	warning	> 80% CPU for 10m
TLSCertExpiryWarning	probe_ssl_earliest_cert_expiry	warning	Expires < 30 days
TLSCertExpiryCritical	probe_ssl_earliest_cert_expiry	critical	Expires < 7 days

Runtime Compatibility

• atlas_* metrics are exposed by 11 Python services via the atlas_observability library
• node_* metrics require node_exporter to be added to the monitoring stack
• probe_ssl_* metrics require blackbox_exporter to be added to the monitoring stack

Prerequisites

• For disk space, memory, and CPU alerts: add node_exporter to docker-compose.monitoring.yml
• For TLS certificate alerts: add blackbox_exporter to docker-compose.monitoring.yml

[Senthil455]

This is a substantial PR adding Prometheus metrics instrumentation across the stack. A few observations:

1. Cross-language consistency looks good — Node.js (prom-client), Python (AtlasMetricsMiddleware), and Go/Fiber (prometheus/client_golang) all appear to implement the same core metrics:

   • atlas_http_requests_total
   • atlas_http_request_duration_seconds
   • atlas_http_requests_in_progress

2. auth-service and api-gateway

   • Excluding /metrics and /health from instrumentation is the correct approach.
   • Path normalization for parameterized routes appears reasonable and should help control label cardinality.

3. Python services

   • AtlasMetricsMiddleware is referenced throughout the diff, but I don't see its implementation or import source included here.

   • Please verify that:

     • The middleware exists in the shared observability package.
     • The import path resolves correctly in all services.
     • The middleware is actually registered in the request pipeline for each service.

   • This is the main area I'd like to double-check before approval.

4. Go services

   • The Fiber middleware implementation looks correct.
   • Using adaptor/v2 for /metrics and collecting request metrics in middleware follows established patterns.

5. Alerting

   • Alert rules have been updated consistently to use the new atlas_ metric namespace.

6. Prometheus configuration

   • The additional scrape targets (live-service and workforce-planning-service) look appropriate.

Overall, the instrumentation approach is well-structured and consistent across the stack. My only significant concern is ensuring the Python AtlasMetricsMiddleware dependency and registration path are present and validated in all affected services.

[DivyashreeR008]

Thanks for the review. The three atlas_* metrics referenced by the alert rules are already defined in the shared atlas_observability library:

• Atlas_http_requests_total (Counter: method, path, status_code)
• Atlas_http_request_duration_seconds (Histogram: method, path, status_code)
• Atlas_http_requests_in_progress (Gauge: method, path)

These are implemented in services/atlas_observability/atlas_observability/metrics_middleware.py, and AtlasMetricsMiddleware is exported via atlas_observability/__init__.py. All 11 Python services already import it using:

from atlas_observability import AtlasMetricsMiddleware

As a result, no code changes are required in this PR. This PR only updates alert.rules.yml; the middleware instrumentation and metric definitions were delivered as part of a previous effort.