feat(jobs): hourly synthetic deploy prober (DEPLOY-001)

[mastermanas805]

Summary

• Adds DeployProbeWorker — every 60 minutes drives a real end-to-end deploy against the prod /deploy/new pipeline (Kaniko build → k8s pod → public-host GET) and pages NR P0 on any leg failure.
• Closes the gap that hid the 2026-05-30 morning truehomie-api stuck-build incident for ~30 minutes until the user reported it.
• Sibling to AUTH-004 (worker#68): same metric / leg / result taxonomy, same audit_log + structured-slog dual surface, same fail-open posture when the probe bearer is unset.

Three legs per tick

1. submit — POST /deploy/new with embedded nginx tarball (in-memory, no fixture dep), name=deploy-probe-hourly, port=80, env=development, redeploy=true. The redeploy flag reuses one persistent probe-app row forever — no deployment-slot accumulation.
2. status — poll GET /deploy/<app_id> every 5s for up to 90s, asserting status flips to healthy. building past 90s and failed are both fail-alerts (worker#65 + fix(deploy_failure_autopsy): capture build logs + emit failure email backstop #66 already flip the row within ~30s).
3. serve — GET https://<app_id>.deployment.instanode.dev/ expecting 200, covering the Ingress / TLS / pod-readiness surface the api's status field doesn't see.

Metrics

• instant_deploy_probe_outcome_total{leg,result} (CounterVec) — lazy: pass/degraded materialise on the first happy tick; fail only appears on real regression.
• instant_deploy_probe_latency_seconds{leg} (HistogramVec) — observation suppressed on DNS/TCP errors so the histogram isn't polluted with 0s timeouts.

Rule-25 surface — companion infra PR

The metric + alert + dashboard + catalog row land in the same PR-time per CLAUDE.md rule 25:

• infra PR (split because cross-repo): https://github.com/InstaNode-dev/infra/pull/new/feat/deploy-probe-hourly (deploy-probe-fail.json NR alert + DeployProbeFail Prom rule + three dashboard tiles mirroring the AUTH-004 layout + two METRICS-CATALOG.md rows).

Coverage block

Symptom:        /deploy/new pipeline broken (api crash, Kaniko gone,
                GHCR auth, Ingress / TLS regression)
Enumeration:    rg -F 'deploy_probe' internal/ (3 files: deploy_probe.go,
                deploy_probe_test.go, workers.go registration)
Sites found:    3
Sites touched:  3
Coverage test:  TestPeriodicJobs_AllCarryUniqueOpts iterates the live
                periodic-job registry — adding the deploy probe
                automatically asserts its UniqueOpts shape.
Live verified:  awaiting operator verification of
                DEPLOY_PROBE_BEARER_TOKEN secret seed + probe-team
                creation (rule 14 + rule 21).

Anti-goals (per brief — all honoured)

• ❌ Don't burn a deploy slot per tick → redeploy=true reuses one row.
• ❌ Don't probe with prod team's token → synthetic probe-team owns the row.
• ❌ Don't run every minute → 60m cadence; the 30m NR alert window guarantees a P0 inside one tick.
• ❌ Don't depend on customer DNS → *.deployment.instanode.dev wildcard is platform-owned.

Test plan

• make gate green locally (build + vet + go test ./... -short -count=1)
• go test ./internal/jobs/ -run 'TestDeployProbe|TestValidateDeployProbeBaseURL' -count=1 -short → ok instant.dev/worker/internal/jobs 1.850s (27 tests pass)
• Hermetic test coverage of all three legs (pass + each fail mode) via httptest.Server mocking the api's /deploy/new + /deploy/:id + public-host surface.
• Operator seeds k8s secret instant-secrets.DEPLOY_PROBE_BEARER_TOKEN (long-lived JWT for synthetic probe team).
• Operator creates the probe team (via /internal/teams or onboarding).
• After CI auto-deploy lands master, verify build-SHA gate per CLAUDE.md rule 14: curl https://api.instanode.dev/healthz | jq .commit_id matches HEAD.
• Verify first hourly tick on prod shows result=pass on all three legs (operator confirmation per rule 21).

Operator follow-ups (not blocking land)

1. Create synthetic probe team; mint long-lived session JWT.
2. Add to instant-secrets: kubectl create secret generic instant-secrets --from-literal=DEPLOY_PROBE_BEARER_TOKEN=<jwt> --dry-run=client -o yaml | kubectl apply -f - (then update the worker Deployment env so the var is picked up at pod start).
3. Apply infra PR sibling (Prom rule + NR alert + dashboard).
4. Confirm first prod tick after the secret rollout produces pass on all three legs.

🤖 Generated with Claude Code