fix: expose /metrics endpoint on all services for Prometheus scraping - #83
Conversation
Problem: Prometheus scraped 14 targets at /metrics but only Java services (payroll, leave via /actuator/prometheus) exposed a metrics endpoint. Node.js (Express), Python (FastAPI), and Go services all returned 404, causing half of all scrape targets to fail. Alert rules referenced http_requests_total which was never emitted, so alerts never fired. Changes: - Python (FastAPI): Added missing AtlasMetricsMiddleware registration to audit-compliance, integration, and security services. Bypassed internal-auth for /metrics on all 11 Python services so Prometheus scrapes aren't rejected with 401. - Node.js (Express): Added prom-client dependency to api-gateway and auth-service. Created metrics middleware tracking request count, duration histogram, and in-progress gauge. Exposed /metrics endpoint returning Prometheus-format metrics. - Go: Added prometheus/client_golang dependency to attendance, lms, and notification services. Added metrics middleware and /metrics endpoint (via promhttp for net/http, via adaptor for Fiber). - Config: Added live-service (8060) and workforce-planning-service (8017) scrape targets to prometheus.yml. - Alert rules: Fixed HighErrorRate alert to use atlas_http_requests_total (the actual metric name) instead of http_requests_total.
There was a problem hiding this comment.
This PR replaces the old HighMemoryUsage alert with more useful latency and infrastructure alerting. A few notes:
-
Atlas_http_requests_total — assumes the metrics from PR #82 are deployed. These two PRs should be merged together or in order (#82 first, then #83).
-
Node-level alerts — disk, memory, CPU, and TLS cert expiry alerts are valuable additions for production monitoring.
-
HighLatencyP99 thresholds — 1.0s (warning) and 2.5s (critical) seem reasonable as starting points.
-
Consider adding a grafana_alerting annotation section or linking these to a dashboard for discoverability. Not blocking.
Looks good, but coordinate merge order with PR #82 since the metric names depend on it.
|
Thanks for the review @Senthil455!
No changes needed from my end. Ready to merge whenever you are. |
Summary
Prometheus is configured to scrape all 14 service targets at
/metrics, but only Java services (payroll, leave via/actuator/prometheus) exposed a metrics endpoint. Node.js (Express), Python (FastAPI), and Go services all returned 404, causing half of all scrape targets to fail.Changes
Python (FastAPI) — 11 services
AtlasMetricsMiddlewareregistration to audit-compliance-service, integration-service, and security-service (they imported it but never calledapp.add_middleware)/metrics(alongside/health) so Prometheus scrapes aren't rejected with 401Node.js (Express) — 2 services
prom-clientdependency with:atlas_http_requests_totalcounteratlas_http_request_duration_secondshistogramatlas_http_requests_in_progressgauge/metricsroute returning Prometheus-format outputGo — 3 services
prometheus/client_golangandgofiber/adaptor/v2dependencies. Added metrics middleware and/metricsroute viapromhttp.Handler()prometheus/client_golangdependency. AddedmetricsMiddlewarewrapping the mux and/metricsroute viapromhttp.Handler()Configuration
live-service:8060andworkforce-planning-service:8017HighErrorRatealert to useatlas_http_requests_total{status_code=~"5.."}(the actual metric name) instead of the non-existenthttp_requests_total{status=~"5.."}Impact
/metricsendpointatlas_http_*)http_requests_totalalert rule that was never triggered is now fixed