fix(deploy_status_reconcile): detect Job-failed silent build crashes

[mastermanas805]

Summary

Fix the silent-deploy-failure bug class hit on 2026-05-30 (user deploy
truehomie-api-36ace884.deployment.instanode.dev sat at building
forever after a kaniko build crashed and the pod was GC'd).

The reconciler previously only queried appsv1.Deployments(ns).Get. When
a build Job hits BackoffLimitExceeded or DeadlineExceeded, the runtime
Deployment is NEVER created (buildImage returns before applyDeployment
runs), so GetDeployment returned NotFound and the row mapped to stopped
— a terminal status that looks to the user like a teardown, with no
autopsy and no failure surface.

This PR adds a second probe: after GetDeployment, the reconciler also
calls BatchV1().Jobs(ns).Get("build-<appID>"). A Job with a
JobFailed condition OR Status.Failed > BackoffLimit is authoritative
— flip the row to failed regardless of what the Deployment says. The
Job object survives its TTLSecondsAfterFinished (5 min in the api)
which is long enough for the 30s reconciler tick to observe it.

The existing in-sweep autopsy capture path fires on the resulting
status transition unchanged. PR2 (separate, for independent rollback)
covers the autopsy log-capture path when the pod is already gone.

Coverage block (rule 17)

Symptom:       deployments.status stuck at 'building' after kaniko
               BackoffLimitExceeded; no autopsy, no failure email.
Enumeration:   rg "GetDeployment|appsv1.Deployments" worker/internal/jobs
Sites found:   1 (deploy_status_reconcile.go:computeNewStatus)
Sites touched: 1
Coverage test: TestDeployStatusReconcile_JobFailedAfterPodGC
               TestDeployStatusReconcile_JobActiveAndPodMissing_StaysBuilding
               TestDeployStatusReconcile_BothNotFound_StaysStopped
               TestDeployStatusReconcile_JobQueryError_FallsThroughToDeployment
               TestJobIsFailed_Matrix
               TestJobFailureReason
               TestK8sDeployStatusClient_GetBuildJob_NotFoundPath
Live verified: PENDING — verify post-merge via rule 14 (curl healthz),
               then trigger a deliberately failing Dockerfile build and
               observe row transition to 'failed' within ~30s.

Rule 25 observability

New Prom counter instant_deploy_job_failed_detected_total{reason}
(labels: BackoffLimitExceeded, DeadlineExceeded, PodFailurePolicy,
plus bounded fallbacks failed_no_reason and backoff_limit_exceeded).
NR alert suggestion documented inline; dashboard tile +
METRICS-CATALOG.md row tracked in the infra repo follow-up.

Test plan

• make gate green locally (matches deploy.yml test step)
• All new unit tests pass: go test ./internal/jobs/... -run 'TestDeployStatusReconcile_Job|TestJobIsFailed|TestJobFailureReason|TestK8sDeployStatusClient_GetBuildJob|TestDeployStatusReconcile_BothNotFound' -count=1
• CI green
• After merge + deploy: verify commit_id on /healthz matches HEAD (rule 14)
• After merge + deploy: enqueue a deliberately failing build, observe row flips to failed within ~30s, autopsy row created

🤖 Generated with Claude Code