fix(deploy_failure_autopsy): capture build logs + emit failure email backstop - #66
Merged
Merged
Conversation
…backstop
The autopsy already wrote a deployment_events row with kind=failure_autopsy,
but in the 2026-05-30 silent-deploy-failure incident:
- deployments.error_message was never updated → CLI / dashboard list
views showed an empty error column.
- No audit_log row was emitted → event_email_forwarder never dispatched
the user-visible failure email (the api's runDeploy normally emits
this, but the api goroutine had already crashed mid-build).
- The autopsy only queried the runtime app pod (label
instant-app-id=<appID>). For a Job-only failure (PR 1's case) the
app pod was never created → last_lines stayed empty even when the
build pod was still alive with the kaniko stderr tail that explains
the failure.
This PR adds three things to captureDeploymentAutopsy:
1. Build-pod fallback: when the app pod yields no logs, list pods
matching label "job-name=build-<appID>" and pull the kaniko log
tail. On success the reason is upgraded from Unknown to BuildFailed.
2. updateDeploymentErrorMessage: stamps "<reason>: <hint snippet>"
onto deployments.error_message when the column is NULL or empty
(the non-clobber guard prevents overwriting a more-specific
api-side error).
3. emitDeployFailedAudit: looks up team_id from deployments, then
INSERTs an audit_log row with kind=deploy.failed so
event_email_forwarder dispatches the failure email. Metadata
includes source="worker_autopsy" so an operator can distinguish
the worker-backstop emit from the api's synchronous one.
Rule 25 observability: new Prom counter
instant_deploy_autopsy_captured_total{outcome} with bounded labels
(logs_captured | logs_unavailable | already_present | audit_emit_failed).
NR alert suggestions documented in metrics.go. All four label families
are primed at process start (metrics_test.go) so the dashboard panel
renders from /metrics scrape #1.
Coverage block:
Symptom: failed deployments had no error_message in the API row
response and the user never received a deploy.failed email.
Enumeration: rg "captureDeploymentAutopsy|deploy.failed|error_message"
worker/internal/jobs (also api/internal/handlers for sources).
Sites found: 1 (deploy_failure_autopsy.go:captureDeploymentAutopsy).
Sites touched: 1 (callers in deploy_status_reconcile.go work unchanged).
Coverage test: TestAutopsy_PodAlive_CapturesLogs,
TestAutopsy_PodGCd_FallsBackToJobEvent,
TestAutopsy_Idempotent,
TestAutopsy_BuildPodFallback,
TestUpdateDeploymentErrorMessage_OnlyUpdatesEmptyColumn,
TestUpdateDeploymentErrorMessage_DBError,
TestFirstSentence,
TestEmitDeployFailedAudit_NoRow / _LookupError /
_NilTeamID / _InsertError / _SummaryTruncation,
TestAutopsyAlreadyPresentWithReason (3 branches),
TestFindBuildPodName (3 branches),
TestAutopsy_NamespaceMismatch_EarlyReturn.
Live verified: PENDING — verify post-merge via rule 14, then trigger a
deliberately failing Dockerfile build and confirm the user
receives the deploy.failed email AND GET /deploy/:id
returns a populated error_message + failure.last_lines.
Pairs with PR fix/deploy-status-reconcile-job-failed (PR 1) — together
they close the silent-deploy-failure bug class. Each ships as a separate
PR for independent rollback per swarm charter.
make gate: green locally (matches deploy.yml test step).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI feedback on the parent commit:
- lint: const autopsyOutcomeAuditEmitFailed was unused. It is now
emitted from the audit-emit error path (paired with the existing
WARN log), which is what the constant always documented but the
caller didn't actually invoke.
- coverage: patch coverage gate (100%) flagged four uncovered
branches in deploy_failure_autopsy.go. Each now has a dedicated test:
* already_present outcome path → TestAutopsy_IdempotentAlreadyPresentBranch
* build-pod fallback reason upgrade Unknown→BuildFailed →
TestAutopsy_BuildPodFallback_UpgradesUnknownToBuildFailed
* updateDeploymentErrorMessage empty-reason fallback →
TestUpdateDeploymentErrorMessage_EmptyReasonFallsBackToUnknown
* audit-emit failure metric increment →
TestAutopsy_AuditEmitFailed_IncrementsCounter
json.Marshal of a map[string]any error path (unreachable for this
shape) was simplified to mirror the orphan_sweep_reconciler's
emitOrphanAudit pattern (_-ignore).
Local diff-cover against origin/master now reports 100% patch coverage
on internal/jobs/deploy_failure_autopsy.go.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mastermanas805
force-pushed
the
fix/deploy-failure-autopsy-log-capture
branch
from
May 30, 2026 11:23
6ccaadf to
56b6174
Compare
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 2 of the 2026-05-30 silent-deploy-failure fix (paired with
#65 / fix/deploy-status-reconcile-job-failed). Closes the autopsy side of
the bug class: the user's deploy got a (now-correctly-detected) status flip
to `failed` via PR 1, but their UI still showed an empty error column and
no email arrived because the autopsy never:
runDeploy normally emits this, but the api goroutine had crashed
mid-build so it never fired),
never created (the modal case for Job-failed deploys).
This PR adds:
Build-pod log fallback. When the app-pod query yields no logs,
list pods matching label `job-name=build-` and pull the
kaniko stderr tail. Reason is upgraded from Unknown → BuildFailed.
error_message stamping. `: ` (first
sentence of the hint, capped at 200 chars) is written to
`deployments.error_message` IFF the column is NULL or empty
(non-clobber guard preserves an api-side specific error).
Audit-log emit. Looks up `team_id` from deployments, then
INSERTs an `audit_log` row with kind=`deploy.failed` and
metadata `source: "worker_autopsy"` so an operator can
distinguish the worker backstop from the api's emit.
Rule 25 observability
New Prom counter `instant_deploy_autopsy_captured_total{outcome}`
with bounded labels: `logs_captured`, `logs_unavailable`,
`already_present`, `audit_emit_failed`. NR alert suggestions
inline. All four label families primed at process start so the
dashboard panel renders from scrape #1.
Coverage block (rule 17)
```
Symptom: failed deployments had no error_message in the API row
and the user never received the deploy.failed email.
Enumeration: rg "captureDeploymentAutopsy|deploy.failed|error_message"
Sites found: 1 (deploy_failure_autopsy.go:captureDeploymentAutopsy)
Sites touched: 1 (callers in deploy_status_reconcile.go unchanged)
Coverage test: TestAutopsy_PodAlive_CapturesLogs
TestAutopsy_PodGCd_FallsBackToJobEvent
TestAutopsy_Idempotent
TestAutopsy_BuildPodFallback
TestUpdateDeploymentErrorMessage_OnlyUpdatesEmptyColumn
TestUpdateDeploymentErrorMessage_DBError
TestFirstSentence
TestEmitDeployFailedAudit_{NoRow,LookupError,NilTeamID,InsertError,SummaryTruncation}
TestAutopsyAlreadyPresentWithReason (3 branches)
TestFindBuildPodName (3 branches)
TestAutopsy_NamespaceMismatch_EarlyReturn
Live verified: PENDING — verify post-merge via rule 14, trigger a
deliberately failing Dockerfile build, confirm user
receives the deploy.failed email AND GET /deploy/:id
returns a populated error_message + failure.last_lines.
```
Test plan
observe (a) email in inbox, (b) error_message populated in API,
(c) failure.last_lines contains kaniko stderr.
🤖 Generated with Claude Code