Skip to content

fix: expose /metrics endpoint on all services for Prometheus scraping - #83

Merged
Senthil455 merged 1 commit into
Senthil455:mainfrom
DivyashreeR008:fix-prometheus-metrics
Jun 17, 2026
Merged

fix: expose /metrics endpoint on all services for Prometheus scraping#83
Senthil455 merged 1 commit into
Senthil455:mainfrom
DivyashreeR008:fix-prometheus-metrics

Conversation

@DivyashreeR008

Copy link
Copy Markdown
Collaborator

Summary

Prometheus is configured to scrape all 14 service targets at /metrics, but only Java services (payroll, leave via /actuator/prometheus) exposed a metrics endpoint. Node.js (Express), Python (FastAPI), and Go services all returned 404, causing half of all scrape targets to fail.

Changes

Python (FastAPI) — 11 services

  • Added missing AtlasMetricsMiddleware registration to audit-compliance-service, integration-service, and security-service (they imported it but never called app.add_middleware)
  • Updated internal-auth middleware in all Python services to bypass /metrics (alongside /health) so Prometheus scrapes aren't rejected with 401

Node.js (Express) — 2 services

  • api-gateway-node & auth-service: Added prom-client dependency with:
    • atlas_http_requests_total counter
    • atlas_http_request_duration_seconds histogram
    • atlas_http_requests_in_progress gauge
    • Default metrics collection
    • /metrics route returning Prometheus-format output
    • path normalization for UUID/numeric route params

Go — 3 services

  • attendance-service & lms-service (Fiber v2): Added prometheus/client_golang and gofiber/adaptor/v2 dependencies. Added metrics middleware and /metrics route via promhttp.Handler()
  • notification-go-service (net/http): Added prometheus/client_golang dependency. Added metricsMiddleware wrapping the mux and /metrics route via promhttp.Handler()

Configuration

  • prometheus.yml: Added missing scrape targets for live-service:8060 and workforce-planning-service:8017
  • alert.rules.yml: Fixed HighErrorRate alert to use atlas_http_requests_total{status_code=~"5.."} (the actual metric name) instead of the non-existent http_requests_total{status=~"5.."}

Impact

  • All 20+ service targets now expose a valid /metrics endpoint
  • Metrics are standardized across all runtimes with consistent naming (atlas_http_*)
  • Alert rules now reference the correct metric names
  • Dashboards for Python and Node.js services will show data
  • The http_requests_total alert rule that was never triggered is now fixed

Problem:
Prometheus scraped 14 targets at /metrics but only Java services
(payroll, leave via /actuator/prometheus) exposed a metrics endpoint.
Node.js (Express), Python (FastAPI), and Go services all returned 404,
causing half of all scrape targets to fail. Alert rules referenced
http_requests_total which was never emitted, so alerts never fired.

Changes:
- Python (FastAPI): Added missing AtlasMetricsMiddleware registration
  to audit-compliance, integration, and security services. Bypassed
  internal-auth for /metrics on all 11 Python services so Prometheus
  scrapes aren't rejected with 401.
- Node.js (Express): Added prom-client dependency to api-gateway and
  auth-service. Created metrics middleware tracking request count,
  duration histogram, and in-progress gauge. Exposed /metrics endpoint
  returning Prometheus-format metrics.
- Go: Added prometheus/client_golang dependency to attendance,
  lms, and notification services. Added metrics middleware and /metrics
  endpoint (via promhttp for net/http, via adaptor for Fiber).
- Config: Added live-service (8060) and workforce-planning-service
  (8017) scrape targets to prometheus.yml.
- Alert rules: Fixed HighErrorRate alert to use atlas_http_requests_total
  (the actual metric name) instead of http_requests_total.

@Senthil455 Senthil455 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR replaces the old HighMemoryUsage alert with more useful latency and infrastructure alerting. A few notes:

  1. Atlas_http_requests_total — assumes the metrics from PR #82 are deployed. These two PRs should be merged together or in order (#82 first, then #83).

  2. Node-level alerts — disk, memory, CPU, and TLS cert expiry alerts are valuable additions for production monitoring.

  3. HighLatencyP99 thresholds — 1.0s (warning) and 2.5s (critical) seem reasonable as starting points.

  4. Consider adding a grafana_alerting annotation section or linking these to a dashboard for discoverability. Not blocking.

Looks good, but coordinate merge order with PR #82 since the metric names depend on it.

@DivyashreeR008

Copy link
Copy Markdown
Collaborator Author

Thanks for the review @Senthil455!

  • The alert rules in this PR only fix the metric name in the existing HighErrorRate rule (from the non-existent http_requests_total to atlas_http_requests_total that this PR introduces). No alerts were removed or replaced.
  • The atlas_http_requests_total metric name is defined within this PR by each service's instrumentation (Python prometheus_client, Node.js prom-client, Go client_golang), so there's no dependency on PR fix: make Prometheus alert rules runtime-agnostic and add comprehensive production alerts #82 for the metric name itself.
  • Agreed on merge order — either order works since both PRs add the same metric name, but merging this one first ensures the alert rule references a metric that actually exists.

No changes needed from my end. Ready to merge whenever you are.

@Senthil455
Senthil455 merged commit a615f96 into Senthil455:main Jun 17, 2026
7 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MEDIUM: Prometheus scrapes 14 targets but only Go/Java services expose metrics

2 participants