feat(relay): add early startup lifecycle logs

[ravarora2]

Why

Early relay failures can currently appear as a container restart without a trustworthy in-process account of whether crypto, structured logging, configuration, relay identity, or the metrics listener failed. Most of those steps happen before the Prometheus exporter exists, so their chronology belongs in logs rather than metrics.

Implements the logs-only early-startup slice of #7238. Post-bind Prometheus exporter supervision is tracked separately in #7284.

What changed

• create a process lifecycle recorder before the Tokio runtime and emit a fixed, versioned JSON schema directly to stderr;
• record started and exactly one terminal event for crypto_init, tracing_init, config_load, key_load, metrics_bind, and the aggregate process_telemetry phase;
• keep every status and reason bounded and suppress raw errors that could contain credentials, keys, URLs, or other secrets;
• return typed metrics-install errors so metrics_bind can be classified without logging raw values, while preserving the existing public metrics::install API;
• document the logs-only evidence contract and add real child-process regressions for success and failure paths.

This PR adds no startup metric families and no dashboard contract. Existing application metrics remain unchanged.

Verification

Exact head: 8faf7526822a119efa035e58b2b3c59aa67fc81d

• cargo fmt --all -- --check
• cargo clippy -p buzz-relay --all-targets -- -D warnings
• relay binary target: 13 passed, 1 PostgreSQL-only test ignored
• real relay child-process lifecycle target: 9 passed
• full relay package library target: 1,023 passed, 89 ignored; the same six media tests failed at crates/buzz-relay/src/api/media.rs:1145 with Sqlx(PoolTimedOut) because local PostgreSQL is unavailable
• three independent exact-head reviews found no correctness, security, compatibility, lifecycle-accounting, logs-only-scope, or test-adequacy finding

All exact-head GitHub CI gates are green, including lint, unit tests, PostgreSQL, relay/backend/desktop integration, both Linux server cross-compiles, Windows/macOS builds, and security checks.

Staging verification

• exact multi-architecture image: dev-sha-8faf7526822a119efa035e58b2b3c59aa67fc81d-run-33708188952-1
• immutable manifest: sha256:26cad28266a6bb0b0e7081eb6091d374e5489f8bb78c475a4a65737dee86cc67
• image workflow: https://github.com/block/buzz/actions/runs/33708188952
• focused staging deployment: https://github.com/squareup/builderbot-platform-core-infrastructure/pull/314
• replacement ReplicaSet buzz-d68764bc7 has two Ready pods with zero restarts
• Datadog received one complete, contiguous sequence 1-12 from each pod; both end with process_telemetry/terminal/succeeded at 3 ms
• queries scoped to the replacement ReplicaSet return no data for the removed buzz_startup_phase_terminal or buzz_startup_phase_duration_seconds families

The experimental Row 7 was removed from the Buzz Startup & Rollout Safety dashboard. This logs-only PR deliberately adds no replacement dashboard row.

---

Update Sep 3, 12:26 ET: Clarified the review boundary: this PR does not close the broader #7238. Later exporter-task termination is pre-existing runtime behavior and is now explicitly tracked in #7284; no production code or staged image changed in this update.

Generated with Codex

[github-actions]

🔐 Codex Security Review

Status: review required for the current range.

The current range is d5a73b9f35feaa81ef9afb13d1ec8943e8ec6ca9...8faf7526822a119efa035e58b2b3c59aa67fc81d.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review 8faf7526822a119efa035e58b2b3c59aa67fc81d to authorize a new review.
Any previous review applies only to its recorded range.

[TheSentinel454]

P1 — post-bind metrics exporter failure remains detached and silent

At exact head 8faf7526822a119efa035e58b2b3c59aa67fc81d, crates/buzz-relay/src/metrics.rs:197-210 spawns the exporter, drops its JoinHandle, and returns success. If the exporter later exits, the relay remains running after reporting metrics_bind/succeeded, with no lifecycle terminal or log. This leaves a silent failure path.

This also leaves the stated contract of #7238 incomplete: that issue explicitly calls for retaining supervision of the exporter lifecycle, while this PR says Closes #7238.

Please either:

1. retain/supervise the exporter task, emit a bounded secret-safe failure if it exits unexpectedly, and add a discriminating production-seam test; or
2. make the deferral explicit by removing Closes #7238 and filing/linking a follow-up issue for exporter supervision.

The existing successful-start and occupied-port tests cover pre-bind behavior, but do not exercise a post-bind exporter exit.

[ravarora2]

🤖 Thanks — confirmed. Dropping Tokio's JoinHandle detaches rather than cancels the exporter, and the real-process scrape test proves the exporter continues serving after try_install returns. However, a later exporter-future terminal is not observed by the relay. That behavior predates this PR, but you are right that it leaves the broader #7238 supervision requirement incomplete.

I took option 2 to keep this PR within its logs-only early-startup scope:

• removed Closes #7238;
• clarified that feat(relay): add early startup lifecycle logs #7258 implements only the logs-only early-startup slice; and
• filed Supervise the Prometheus exporter after startup #7284 for retained exporter ownership, a bounded secret-safe runtime failure, owned shutdown with an independent deadline, teardown, and production-seam post-bind-exit regressions.

No production code or staged image changed as part of this scope correction.

[wpfleger96]

One blocking correctness gap remains. The bounded schema, secret-safe reasons, early stderr sink, exact terminal accounting, and child-process coverage are otherwise solid.

Quality scores:

• Minimalism: 9/10 — the schema and typed classifications are tightly bounded.
• Elegance: 9/10 — phase ownership and aggregate accounting are clear and fit the startup path.
• Correctness: 8/10 — bind/build/conflict handling is covered, but the exporter lifecycle required by #7238 is still detached.

CI is green at exact head 8faf7526822a119efa035e58b2b3c59aa67fc81d.

[wpfleger96]

IMPORTANT — Correctness: This still drops the exporter JoinHandle immediately after reporting metrics_bind/succeeded. If this future later returns Err or panics, the relay keeps running with a dead metrics endpoint and no in-process failure record. That leaves #7238’s explicit exporter-supervision requirement incomplete. Please retain and supervise the task, surface an unexpected exit with a bounded secret-safe reason, and add a production-seam regression that forces a post-bind exit. If supervision is intentionally deferred, this PR should stop closing #7238 and link a follow-up issue instead.

[wpfleger96]

Independent live verification at the same exact head confirms this failure mode through the production API seam:

• Built buzz-relay at 8faf7526822a119efa035e58b2b3c59aa67fc81d (release artifact SHA-256 d98e1dead1086cd126437c195e2d5b971f742e6d53523679af80ea7a1a9b7c40).
• Started BootTracker, ran required MetricsBind, called metrics::try_install(), and successfully scraped the actual bound HTTP exporter.
• Dropped the exporter’s Tokio runtime to force a post-bind task exit while leaving the process alive. The endpoint then refused connections, but the process remained alive.
• Lifecycle evidence remained limited to metrics_bind and aggregate process_telemetry terminal records with status: "succeeded"; no terminal or degradation record reported the exporter’s disappearance.

Observed probe result:

PROBE bound_and_scrapeable=true port=65429
PROBE child_alive_after_exporter_exit=true connect_after_runtime_drop=Err(... ConnectionRefused ...)

This independently confirms that the detached JoinHandle leaves stale, materially false process-local lifecycle evidence after metrics loss. External scrape alerts can detect absence, but they do not supervise, restart, terminate, or correct lifecycle state. The IMPORTANT correctness severity and required fix above stand. The probe worktree, listener, and process were cleaned up afterward.

[ravarora2]

🤖 Thanks for the independent live reproduction. I agree the post-bind exporter terminal is currently unobserved.

This PR is taking the deferral option from the original comment. The current PR body no longer closes #7238; it explicitly scopes #7258 to the logs-only early-startup slice and links #7284. #7284 tracks retained exporter ownership, bounded secret-safe runtime-failure reporting, owned drain/teardown under an independent deadline, and the post-bind production-seam regressions.

The existing metrics_bind and process_telemetry terminals remain immutable startup receipts; #7284 must report the later runtime failure without emitting a second startup terminal. No production code or staged image changed for this scope correction.

Please re-review the corrected scope when you can.

[wpfleger96]

I re-reviewed the corrected scope. Removing the closing reference to #7238 and explicitly tracking post-bind exporter supervision in #7284 satisfies the deferral option from my original finding. The detached exporter remains a real pre-existing runtime gap, but it is no longer an unfulfilled claim of this logs-only PR, so this blocker is resolved.

[wpfleger96]

The scope correction resolves my requested change: this PR now explicitly delivers only the logs-only early-startup slice, does not close #7238, and tracks post-bind exporter supervision in #7284. The bounded schema, secret-safe reasons, early stderr sink, exact startup terminal accounting, and child-process coverage meet that narrower contract.

Quality scores:

• Minimalism: 9/10 — the fixed schema and typed classifications stay tightly bounded.
• Elegance: 9/10 — phase ownership and aggregate accounting are clear and fit the startup path.
• Correctness: 9/10 — the implementation and tests cover the stated logs-only startup contract; the independently reproduced post-bind runtime gap is explicitly deferred to #7284 rather than claimed here.

All applicable CI checks are green at exact head 8faf7526822a119efa035e58b2b3c59aa67fc81d.