End-to-End OpenTelemetry: Distributed Tracing with W3C Propagation, Log/Trace Correlation, RED Metrics with Exemplars, <5% Overhead

[Christopherdominic]

Summary

Observability is fragmented: Prometheus counters exist (src/metrics/), tracing is initialized, there is a telemetry/ module and a monitoring/ directory — but there is no distributed tracing export, no trace context propagation, and no correlation between traces, logs, and metrics. When a tip request is slow, nothing connects the HTTP span to the SQL queries (db/slow_query_logger.rs logs in isolation), the Redis calls, the outbound webhook, and the Horizon verification that composed it.

Task

1. Wire OpenTelemetry end-to-end: tracing-opentelemetry + OTLP exporter (endpoint/protocol from typed config, gracefully disabled when unset), resource attributes (service name/version/instance), and parent-based + ratio tail-aware sampling configuration.
2. Context propagation: extract/inject W3C traceparent on inbound requests and all outbound calls (reqwest to Horizon, webhook deliveries, queue jobs — spans must link across the async job boundary via span links).
3. Instrument the critical path with meaningful spans and attributes: request span (route template, status, principal id — never PII/secrets in attributes), SQLx query spans (statement summary, rows), Redis command spans, webhook delivery spans, job execution spans.
4. Log correlation: every structured log line carries trace_id/span_id; slow-query logs join the active span instead of logging standalone.
5. Metrics with exemplars: RED metrics (rate/errors/duration histogram) per route template, exemplars linking histogram buckets to trace IDs where the Prometheus setup allows.
6. Ship the local stack: extend compose.yml with an OTLP collector + Jaeger (or Grafana Tempo) profile, plus a provisioned Grafana dashboard JSON in monitoring/ covering RED per route, DB latency, webhook success rate, and queue lag.
7. Overhead guard: benchmark request throughput with tracing enabled vs disabled (benches/), demonstrate <5% overhead at the default sampling rate.
8. Tests: propagation round-trip (inbound traceparent → outbound request carries child context), job-boundary span linkage, and log/trace ID correlation.

Acceptance criteria

• A single trace shows HTTP → SQL → Redis → webhook/job spans for a tip request in Jaeger/Tempo (screenshot this)
• Logs carry trace IDs; slow-query log entries are span-correlated
• RED metrics per route with exemplars exposed on /metrics
• <5% measured overhead at default sampling; exporter cleanly disabled when unconfigured
• All checks pass

PR requirements (mandatory)

• ✅ Your PR must pass all checks — PRs with failing or skipped checks will not be merged.
• 📸 You must attach a screenshot in the PR description demonstrating the result (for this issue: the end-to-end trace waterfall in Jaeger/Tempo and the Grafana dashboard).
• 🔗 You must link this issue number in your PR description (e.g. Closes #<issue-number>). PRs without a linked issue will not be reviewed.

[k-deejah]

Hi, I can work this issue

[grantfox-oss]

🦊 GrantFox — @k-deejah has been assigned to this issue as part of the Official Campaign | FWC26 campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #346)
2. Your PR will be reviewed by the Bonizozo maintainers

Good luck! Track your progress on GrantFox.