Skip to content

fix(deploy_failure_autopsy): capture build logs + emit failure email backstop - #66

Merged
mastermanas805 merged 4 commits into
masterfrom
fix/deploy-failure-autopsy-log-capture
May 30, 2026
Merged

fix(deploy_failure_autopsy): capture build logs + emit failure email backstop#66
mastermanas805 merged 4 commits into
masterfrom
fix/deploy-failure-autopsy-log-capture

Conversation

@mastermanas805

Copy link
Copy Markdown
Member

Summary

PR 2 of the 2026-05-30 silent-deploy-failure fix (paired with
#65 / fix/deploy-status-reconcile-job-failed). Closes the autopsy side of
the bug class: the user's deploy got a (now-correctly-detected) status flip
to `failed` via PR 1, but their UI still showed an empty error column and
no email arrived because the autopsy never:

  • updated `deployments.error_message` with a human-readable cause,
  • emitted an `audit_log` row of kind=`deploy.failed` (the api's
    runDeploy normally emits this, but the api goroutine had crashed
    mid-build so it never fired),
  • fell back to the build pod's logs when the runtime app pod was
    never created (the modal case for Job-failed deploys).

This PR adds:

  1. Build-pod log fallback. When the app-pod query yields no logs,
    list pods matching label `job-name=build-` and pull the
    kaniko stderr tail. Reason is upgraded from Unknown → BuildFailed.

  2. error_message stamping. `: ` (first
    sentence of the hint, capped at 200 chars) is written to
    `deployments.error_message` IFF the column is NULL or empty
    (non-clobber guard preserves an api-side specific error).

  3. Audit-log emit. Looks up `team_id` from deployments, then
    INSERTs an `audit_log` row with kind=`deploy.failed` and
    metadata `source: "worker_autopsy"` so an operator can
    distinguish the worker backstop from the api's emit.

Rule 25 observability

New Prom counter `instant_deploy_autopsy_captured_total{outcome}`
with bounded labels: `logs_captured`, `logs_unavailable`,
`already_present`, `audit_emit_failed`. NR alert suggestions
inline. All four label families primed at process start so the
dashboard panel renders from scrape #1.

Coverage block (rule 17)

```
Symptom: failed deployments had no error_message in the API row
and the user never received the deploy.failed email.
Enumeration: rg "captureDeploymentAutopsy|deploy.failed|error_message"
Sites found: 1 (deploy_failure_autopsy.go:captureDeploymentAutopsy)
Sites touched: 1 (callers in deploy_status_reconcile.go unchanged)
Coverage test: TestAutopsy_PodAlive_CapturesLogs
TestAutopsy_PodGCd_FallsBackToJobEvent
TestAutopsy_Idempotent
TestAutopsy_BuildPodFallback
TestUpdateDeploymentErrorMessage_OnlyUpdatesEmptyColumn
TestUpdateDeploymentErrorMessage_DBError
TestFirstSentence
TestEmitDeployFailedAudit_{NoRow,LookupError,NilTeamID,InsertError,SummaryTruncation}
TestAutopsyAlreadyPresentWithReason (3 branches)
TestFindBuildPodName (3 branches)
TestAutopsy_NamespaceMismatch_EarlyReturn
Live verified: PENDING — verify post-merge via rule 14, trigger a
deliberately failing Dockerfile build, confirm user
receives the deploy.failed email AND GET /deploy/:id
returns a populated error_message + failure.last_lines.
```

Test plan

  • `make gate` green locally
  • All new tests pass
  • CI green
  • After merge + deploy: rule 14 `/healthz` SHA gate
  • After merge + deploy: enqueue a deliberately failing build,
    observe (a) email in inbox, (b) error_message populated in API,
    (c) failure.last_lines contains kaniko stderr.

🤖 Generated with Claude Code

mastermanas805 and others added 2 commits May 30, 2026 16:53
…backstop

The autopsy already wrote a deployment_events row with kind=failure_autopsy,
but in the 2026-05-30 silent-deploy-failure incident:

  - deployments.error_message was never updated → CLI / dashboard list
    views showed an empty error column.
  - No audit_log row was emitted → event_email_forwarder never dispatched
    the user-visible failure email (the api's runDeploy normally emits
    this, but the api goroutine had already crashed mid-build).
  - The autopsy only queried the runtime app pod (label
    instant-app-id=<appID>). For a Job-only failure (PR 1's case) the
    app pod was never created → last_lines stayed empty even when the
    build pod was still alive with the kaniko stderr tail that explains
    the failure.

This PR adds three things to captureDeploymentAutopsy:

  1. Build-pod fallback: when the app pod yields no logs, list pods
     matching label "job-name=build-<appID>" and pull the kaniko log
     tail. On success the reason is upgraded from Unknown to BuildFailed.

  2. updateDeploymentErrorMessage: stamps "<reason>: <hint snippet>"
     onto deployments.error_message when the column is NULL or empty
     (the non-clobber guard prevents overwriting a more-specific
     api-side error).

  3. emitDeployFailedAudit: looks up team_id from deployments, then
     INSERTs an audit_log row with kind=deploy.failed so
     event_email_forwarder dispatches the failure email. Metadata
     includes source="worker_autopsy" so an operator can distinguish
     the worker-backstop emit from the api's synchronous one.

Rule 25 observability: new Prom counter
instant_deploy_autopsy_captured_total{outcome} with bounded labels
(logs_captured | logs_unavailable | already_present | audit_emit_failed).
NR alert suggestions documented in metrics.go. All four label families
are primed at process start (metrics_test.go) so the dashboard panel
renders from /metrics scrape #1.

Coverage block:
  Symptom:       failed deployments had no error_message in the API row
                 response and the user never received a deploy.failed email.
  Enumeration:   rg "captureDeploymentAutopsy|deploy.failed|error_message"
                 worker/internal/jobs (also api/internal/handlers for sources).
  Sites found:   1 (deploy_failure_autopsy.go:captureDeploymentAutopsy).
  Sites touched: 1 (callers in deploy_status_reconcile.go work unchanged).
  Coverage test: TestAutopsy_PodAlive_CapturesLogs,
                 TestAutopsy_PodGCd_FallsBackToJobEvent,
                 TestAutopsy_Idempotent,
                 TestAutopsy_BuildPodFallback,
                 TestUpdateDeploymentErrorMessage_OnlyUpdatesEmptyColumn,
                 TestUpdateDeploymentErrorMessage_DBError,
                 TestFirstSentence,
                 TestEmitDeployFailedAudit_NoRow / _LookupError /
                 _NilTeamID / _InsertError / _SummaryTruncation,
                 TestAutopsyAlreadyPresentWithReason (3 branches),
                 TestFindBuildPodName (3 branches),
                 TestAutopsy_NamespaceMismatch_EarlyReturn.
  Live verified: PENDING — verify post-merge via rule 14, then trigger a
                 deliberately failing Dockerfile build and confirm the user
                 receives the deploy.failed email AND GET /deploy/:id
                 returns a populated error_message + failure.last_lines.

Pairs with PR fix/deploy-status-reconcile-job-failed (PR 1) — together
they close the silent-deploy-failure bug class. Each ships as a separate
PR for independent rollback per swarm charter.

make gate: green locally (matches deploy.yml test step).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI feedback on the parent commit:

- lint: const autopsyOutcomeAuditEmitFailed was unused. It is now
  emitted from the audit-emit error path (paired with the existing
  WARN log), which is what the constant always documented but the
  caller didn't actually invoke.
- coverage: patch coverage gate (100%) flagged four uncovered
  branches in deploy_failure_autopsy.go. Each now has a dedicated test:
    * already_present outcome path → TestAutopsy_IdempotentAlreadyPresentBranch
    * build-pod fallback reason upgrade Unknown→BuildFailed →
      TestAutopsy_BuildPodFallback_UpgradesUnknownToBuildFailed
    * updateDeploymentErrorMessage empty-reason fallback →
      TestUpdateDeploymentErrorMessage_EmptyReasonFallsBackToUnknown
    * audit-emit failure metric increment →
      TestAutopsy_AuditEmitFailed_IncrementsCounter

  json.Marshal of a map[string]any error path (unreachable for this
  shape) was simplified to mirror the orphan_sweep_reconciler's
  emitOrphanAudit pattern (_-ignore).

Local diff-cover against origin/master now reports 100% patch coverage
on internal/jobs/deploy_failure_autopsy.go.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@mastermanas805
mastermanas805 force-pushed the fix/deploy-failure-autopsy-log-capture branch from 6ccaadf to 56b6174 Compare May 30, 2026 11:23
@mastermanas805
mastermanas805 merged commit 49ad37b into master May 30, 2026
11 checks passed
@mastermanas805
mastermanas805 deleted the fix/deploy-failure-autopsy-log-capture branch May 30, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant