Skip to content

Make OTel tracing opt-in instead of always-on - #13

Merged
tompscanlan merged 1 commit into
mainfrom
fix/gate-otel-tracing
Jul 29, 2026
Merged

Make OTel tracing opt-in instead of always-on#13
tompscanlan merged 1 commit into
mainfrom
fix/gate-otel-tracing

Conversation

@tompscanlan

Copy link
Copy Markdown
Contributor

InitTracing defaulted OTEL_EXPORTER_OTLP_ENDPOINT to localhost:4318:

endpoint := getEnvOrDefault("OTEL_EXPORTER_OTLP_ENDPOINT", "localhost:4318")

So unsetting the variable did not disable tracing — it silently retargeted the exporter at localhost, and the batch processor logged export failures on a loop. Combined with InitTracing being called unconditionally from both cmd/api and cmd/consumer, there was no way to turn tracing off from configuration at all.

That became a real problem when we retired our collector: the service kept retrying into the void and filling logs with connection errors.

Change

Gate initialization on two conditions, matching how our Node services already behave so the whole fleet has one consistent switch:

  1. ENABLE_TRACING must be exactly "true"
  2. OTEL_EXPORTER_OTLP_ENDPOINT must be set (no default)

Either gate failing returns a no-op shutdown and leaves the global TracerProvider untouched. Callers always defer the returned function, so it is never nil.

Tests

The previous suite used bare os.Setenv/os.Unsetenv, which leaked state between tests — TestInitTracing_DefaultValues unset both variables and never restored them, so the two tests after it ran against whatever was left over. That is also why gating the behaviour broke TestInitTracing_CreatesSpans: it had been relying on ambient env from earlier tests.

Switched to t.Setenv for automatic per-test restore, and added coverage for:

  • both disabled paths (no ENABLE_TRACING, no endpoint)
  • non-"true" values (false, 1, yes, TRUE) — only exact "true" enables
  • the http:// scheme stripping that the deployment config actually exercises

go vet clean; full suite passes (templ generate first — *_templ.go is gitignored, so a fresh clone needs it before go build).

InitTracing defaulted OTEL_EXPORTER_OTLP_ENDPOINT to "localhost:4318", so
unsetting the variable did not disable tracing -- it silently retargeted the
exporter at localhost and the batch processor logged export failures on a
loop. There was no way to turn tracing off from configuration at all, which
became a problem when we retired our collector: the service kept retrying
into the void and filling logs with connection errors.

Gate initialization on two conditions, matching how our Node services
(openmeet-api, bsky-event-processor, bsky-firehose-consumer) already work, so
the whole fleet has one consistent switch:

  1. ENABLE_TRACING must be exactly "true"
  2. OTEL_EXPORTER_OTLP_ENDPOINT must be set (no default)

Either gate failing returns a no-op shutdown and leaves the global
TracerProvider untouched. Callers always defer the returned function, so it
is never nil.

Tests: the previous suite used bare os.Setenv/os.Unsetenv, which leaked state
between tests -- TestInitTracing_DefaultValues unset both variables and never
restored them, so the two tests after it ran against whatever was left over.
Switched to t.Setenv for automatic per-test restore, and added coverage for
both disabled paths, for non-"true" values, and for the http:// scheme
stripping that the deployment config actually exercises.
@tompscanlan
tompscanlan merged commit 60e4b35 into main Jul 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant