Cross-cutting: No distributed tracing links a donation across client, API, event store, job queue and chain

[Emmy123222]

Goal this serves — Diagnosing donation failures — radical transparency requires knowing what happened

Why this matters

When a donor reports that their donation failed, there is currently no way to reconstruct what happened. The attempt crosses a browser or phone, an HTTP handler, the event store, a background job and Horizon, and nothing carries an identifier through those hops. Support means correlating by timestamp across separate logs and hoping.

This matters more here than in most systems because donations can span days — the offline queue on mobile and the record-failure recovery on web both mean one logical donation may involve several sessions.

Evidence

There is a structured logger at backend/src/utils/logger.js that reads a correlation identifier from async storage, and the API sets an X-Correlation-ID response header. But the identifier does not originate at the client, is not propagated into background jobs, and is not recorded alongside the transaction hash.

No tracing library is present anywhere: a search for OpenTelemetry or an equivalent across backend/src returns nothing. Spans, timings and cross-service causality do not exist.

Why this is hard

The asynchronous boundary is the whole problem. Propagating a header through synchronous HTTP is routine. Carrying causality into an event appended now and projected later, or into a job retried hours afterwards, is not — and that is exactly where donations get lost.

The chain is outside the trace. Horizon submission is the one hop that cannot be instrumented. Linking a trace to a transaction hash, and back, is the only way to close the loop.

Sampling versus completeness. Tracing everything is expensive; sampling means the failure you need is often the one not sampled. Donation flows likely warrant complete capture while browse traffic does not.

Nothing may leak. Traces must never carry secret keys, signed envelopes or donor personal data, and that has to be enforced structurally rather than by reviewer vigilance.

Suggested approach

Generate the identifier at the client so the trace begins where the donation does, propagate it as a header, attach it to event metadata and job payloads, and store it beside the transaction hash so a trace can be found from a chain record and vice versa.

Instrument spans across the request, the projection and the job execution. Decide sampling per route rather than globally.

Enforce redaction at the exporter, and prove it with a test that asserts key material cannot appear in an emitted span.

Acceptance criteria

• A correlation identifier originates at the client and survives into background jobs and later-session retries.
• Spans cover the HTTP request, the event append, the projection and the job execution, with causal links intact.
• A trace can be found from a transaction hash, and a transaction hash from a trace.
• Sampling is configured per route, with donation flows captured completely.
• Redaction is enforced at the exporter and proven by a test asserting key material and signed envelopes cannot appear.
• A documented procedure retrieves the full history of one donation attempt from a single identifier.
• The 100 remaining console.* calls on the donation path are migrated to the structured logger.
• Overhead is measured and recorded against a baseline.

Scope

Roughly 5,000–7,000 lines, including tests.

Relevant files

• backend/src/utils/logger.js
• backend/src/server.js
• backend/src/eventSourcing/eventStore.js
• backend/src/services/summaryQueue.js
• frontend/lib/api.ts