feat(jobs): hourly synthetic deploy prober (DEPLOY-001) (#69)

* feat(jobs): hourly synthetic deploy prober (DEPLOY-001)

Drives a real end-to-end deploy against the prod /deploy/new pipeline
every 60 minutes (POST /deploy/new with redeploy=true → Kaniko build +
k8s rollout → status poll until healthy → public-host GET expecting 200)
and pages NR P0 on any leg failure. Closes the gap that hid the
2026-05-30 morning truehomie-api stuck-build incident for ~30 minutes
until the user reported it. Sibling to AUTH-004 (worker#68): same
metric/leg/result taxonomy, same audit_log + structured-slog dual
surface, same fail-open posture when the probe bearer is unset.

Three legs per tick:
  1. submit — POST /deploy/new with embedded nginx tarball (in-memory,
     no fixture dep), name=deploy-probe-hourly, port=80, env=development,
     redeploy=true. The redeploy flag reuses one persistent probe-app
     row forever — no deployment-slot accumulation.
  2. status — poll GET /deploy/<app_id> every 5s for up to 90s,
     asserting status flips to "healthy". `building` past 90s and
     `failed` are both fail-alerts.
  3. serve — GET https://<app_id>.deployment.instanode.dev/ expecting
     200, covering the Ingress / TLS / pod-readiness surface the api's
     status field doesn't see.

Metric: instant_deploy_probe_outcome_total{leg,result} +
instant_deploy_probe_latency_seconds{leg} (lazy CounterVec +
HistogramVec). Latency observation suppressed on DNS/TCP errors so
the histogram isn't polluted with 0s timeouts.

Audit: kind=deploy_probe_failed, actor='system:deploy_probe',
team_id=NULL (platform-level).

Schedule: every 60m on the reconcile queue (UniqueOpts guards a
replicas:2 cluster from double-firing). RunOnStart=false — the leg-1
submit is heavyweight (~30s of cluster work per tick) so a worker
restart inside the hour doesn't add useful signal beyond the
previous tick's metric.

Anti-goals (per brief):
  - No DELETE: redeploy=true reuses one app row across ticks.
  - Synthetic probe team: DEPLOY_PROBE_BEARER_TOKEN owns the row,
    not a real customer.
  - 60m cadence: a stuck build auto-flips within 30s (worker#65) so
    hourly is plenty; 5m would 12x cluster traffic.
  - Wildcard *.deployment.instanode.dev is platform-owned (no
    customer DNS dependency).

Coverage:
  Symptom:       /deploy/new pipeline broken (api crash, Kaniko gone,
                 GHCR auth, Ingress / TLS regression)
  Enumeration:   rg -F 'deploy_probe' internal/   (3 files; all
                 deploy_probe.go + test + registration in workers.go)
  Sites found:   3
  Sites touched: 3
  Coverage test: TestPeriodicJobs_AllCarryUniqueOpts iterates the
                 live periodic-job registry — adding the deploy
                 probe automatically asserts its UniqueOpts shape.
  Live verified: awaiting operator verification of
                 DEPLOY_PROBE_BEARER_TOKEN secret seed + probe-team
                 creation (rule 14 + rule 21). PR companion infra/ +
                 metric/alert/dashboard surface lands same-time per
                 rule 25.

Operator follow-ups (not blocking PR land):
  - Create synthetic probe team via /internal/teams or onboarding;
    mint a long-lived JWT, store as k8s secret instant-secrets
    DEPLOY_PROBE_BEARER_TOKEN.
  - Operator confirms first hourly tick on prod shows result=pass
    on all three legs after the rollout (build-SHA gate per rule 14).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(deploy_probe): drop unused deployProbeTestClient helper (lint)

* test(deploy_probe): white-box + injectable budgets → 100% patch coverage

Two follow-up cleanups so the CI patch-coverage gate at 100% passes:

  - Make `*Budget` / `pollInterval` injectable on DeployProbeWorker
    (zero = use the package-level constants — production behaviour
    unchanged). The four unexported fields let internal tests drive
    the degraded-latency + after-deadline + ctx-cancelled-during-poll
    branches inside a second of wall-clock instead of needing the real
    30s / 90s budgets.

  - Decouple serveBudget from the request's ctx timeout (matches the
    auth_probe.legEmailStart pattern). Today the leg-3 request was
    cancelled by servCtx the moment its budget elapsed, so the
    degraded-latency branch on `latency > serveBudget` was provably
    unreachable in production (a slow-but-working serve always landed
    in http_error). Now the request uses the http client's hard
    timeout (deployProbeHTTPTimeout = 120s) and the budget is a pure
    post-hoc latency assertion → degraded fires correctly on a slow
    serve that completed before the hard ceiling.

  - Drop defensive `error` returns from `buildDeployProbeMultipart` +
    `buildDeployProbeNginxTarball` — both write to in-memory
    bytes.Buffer writers which physically cannot fail. Same posture
    as auth_probe.legEmailStart's `_ = json.Marshal(...)`. Removes
    the unreachable error branches that were the modal
    patch-coverage gap.

  - Add `deploy_probe_internal_test.go` (jobs package) with white-box
    tests covering: tarball roundtrip, multipart shape, recordLeg's
    three branches, legStatus ctx-cancelled-at-top-of-loop +
    budget-elapsed + poll-interval-sleep-completes, legSubmit
    degraded-latency, legServe degraded-latency + http_error +
    build_request, fetchDeployStatus build_request + http_error,
    effective-budget zero-fallback + override, and the
    CheckRedirect closure on the default client.

Coverage (locally measured against origin/master):
  internal/jobs/deploy_probe.go (100%)
  internal/jobs/workers.go       (100%)
  Total: 437 lines, 0 missing — gate passes.

No production behaviour change beyond the serveBudget decoupling
documented above. `make gate` green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: mastermanas805

internal/config/config.go

@@ -140,6 +140,17 @@ type Config struct {
 	AuthProbeReturnTo    string // AUTH_PROBE_RETURN_TO — must be on api's allow-list
 	AuthProbeOrigin      string // AUTH_PROBE_ORIGIN — must match api CORS allow-list
 	AuthProbeBearerToken string // AUTH_PROBE_BEARER_TOKEN — probe-account session JWT
+
+	// Hourly synthetic deploy prober — drives a real end-to-end deploy
+	// against the prod /deploy/new pipeline every 60 minutes so the
+	// next regression in Kaniko / k8s / Ingress / TLS pages within the
+	// 30-minute alert window. BearerToken is required for the prober to
+	// run; empty causes every leg to report result="degraded" (config
+	// drift, not outage). BaseURL + DeployHost fall back to the production
+	// hosts inside jobs.DeployProbeConfig.Defaults().
+	DeployProbeBaseURL     string // DEPLOY_PROBE_BASE_URL — default https://api.instanode.dev
+	DeployProbeDeployHost  string // DEPLOY_PROBE_DEPLOY_HOST — default deployment.instanode.dev
+	DeployProbeBearerToken string // DEPLOY_PROBE_BEARER_TOKEN — probe-team session JWT (required)
 }
 
 // ErrMissingConfig is returned when a required env var is absent.
@@ -234,6 +245,14 @@ func Load() *Config {
 		AuthProbeReturnTo:    os.Getenv("AUTH_PROBE_RETURN_TO"),
 		AuthProbeOrigin:      os.Getenv("AUTH_PROBE_ORIGIN"),
 		AuthProbeBearerToken: os.Getenv("AUTH_PROBE_BEARER_TOKEN"),
+
+		// Hourly synthetic deploy prober — all optional; defaults applied
+		// inside jobs.DeployProbeConfig.Defaults() so a missing env var
+		// still runs against prod. Empty BearerToken keeps the prober
+		// configured-off (degraded outcomes only, no fail alerts).
+		DeployProbeBaseURL:     os.Getenv("DEPLOY_PROBE_BASE_URL"),
+		DeployProbeDeployHost:  os.Getenv("DEPLOY_PROBE_DEPLOY_HOST"),
+		DeployProbeBearerToken: os.Getenv("DEPLOY_PROBE_BEARER_TOKEN"),
 	}
 
 	// Fall back to the shared object-store bucket when the operator hasn't