fix(worker): pg_dump missing — switch base to postgres:16-alpine (P1) - #72
Merged
Merged
Conversation
The hourly synthetic deploy prober (worker#69) wedged on its first tick
in prod with `404 no_existing_deployment_to_redeploy` — it POSTs with
`redeploy=true` from tick 1, but the persistent `deploy-probe-hourly`
app row doesn't exist yet, so the api's typed-error path correctly
refused. The probe was DOA until the operator manually bootstrapped.
Fix: legSubmit detects the canonical error code on a 404 response,
logs `jobs.deploy_probe.bootstrap_retry`, and transparently retries
ONCE without `redeploy=true` (create semantics). The retry's outcome
is reported as `result=bootstrap` on
`instant_deploy_probe_outcome_total{leg=submit}` — distinct from
`pass` so the dashboard sees the one-time self-heal as its own event.
Subsequent ticks find the row, get a 2xx, and report `pass`.
Anti-design guardrails:
- ONLY the canonical `no_existing_deployment_to_redeploy` error code
triggers the retry. A non-canonical 404 (auth misroute, future
api-side regression that drops `error`) still fails the leg with
audit_log + ERROR slog — never mask a real outage as a self-heal.
- Bootstrap counts as a successful leg-1 for downstream leg dispatch
so status + serve run on the freshly-bootstrapped row.
- `bootstrap` is NOT a degraded/fail; recordLeg logs at INFO, skips
the audit_log INSERT, and the alert NRQL keys only on `result=fail`.
Coverage block (rule 17):
Symptom: 404 no_existing_deployment_to_redeploy on first probe tick
Enumeration: rg -F 'no_existing_deployment_to_redeploy' (api + worker)
Sites found: 2 in api/internal/handlers/deploy.go (sql.ErrNoRows
and wrong-team defence-in-depth), 0 in worker pre-fix
Sites touched: 1 in worker (new deployProbeRedeployMissingCode const
consumed by legSubmit retry guard); api sites untouched
(already correct typed-error contract per api#206)
Coverage test: TestDeployProbe_Bootstrap_FirstTick404RetriesAsCreate
+ TestDeployProbe_Bootstrap_NonCanonical404StillFails
+ TestBuildDeployProbeMultipart_BootstrapShape
Live verified: pending merge + deploy; gated on rule 14 SHA check post-rollout
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P1 incident 2026-05-30: customer_backup_runner shells out to pg_dump (internal/jobs/customer_backup_runner.go:120). The previous distroless/static-debian12 base has no shell + no pg_dump, so every Pro+ tier customer's scheduled backup has been failing silently with "exec: pg_dump: executable file not found in $PATH" — data-loss risk. Fix: - Runtime base → postgres:16-alpine (matches the customer-pg server version per infra/k8s/postgres-customers.yaml). Image grows ~280MB but correctness > size here. Worker binary is CGO_ENABLED=0 so it runs unmodified on alpine. - Add CI smoke-check in deploy.yml that probes the built image for pg_dump and fails the deploy hard if the binary regresses (or major version < 16). Closes the regression gate that would have caught this the day distroless was introduced. Locally verified: docker run --rm --entrypoint sh worker-test:local -c 'pg_dump --version' → pg_dump (PostgreSQL) 16.13 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P1 incident: 2026-05-30 overnight
Customer backups have been silently failing with
exec: pg_dump: executable file not found in $PATH. Every Pro+ tier customer's scheduled backup broken — data-loss risk.Root cause
worker/internal/jobs/customer_backup_runner.go:120shells out to pg_dump. The previousdistroless/static-debian12base has no shell + no pg_dump.Fix
postgres:16-alpine(matches customer-pg server perinfra/k8s/postgres-customers.yaml). +~280MB image size; correctness > size.Local verify
Coverage block (rule 17)
grep -rn 'exec.Command.*pg_dump' worker/internal/jobs/→ 1 site