Skip to content

Commit 776c414

Browse files
authored
fix(review): persist AI review diagnostics into cache metadata on degraded runs (#9447)
* fix(review): persist AI review diagnostics into cache metadata on degraded runs `ai_review_cache.metadata_json` records `inconclusive` — that a review degraded, but never why. The per-attempt `AiReviewDiagnostic` values that carry the answer were only ever sent to PostHog as a capture property, so the 2026-07-27 investigation into missing review summaries could not distinguish the failure classes from the database or the container logs, and reaching PostHog needs an interactive OAuth an incident responder may not have. Persist the compact `model#attempt:status[:error]` strings alongside `inconclusive`, so `missing_assessment` (the model returned no narrative), `unparseable_output`/`empty_output` (it returned something parseModelReview could not read) and `provider_error` are separable at read time — three different bugs with three different fixes. Written on the degraded paths only, never on the healthy one: every byte lands in a D1 row and this database hit its 10 GB ceiling on 2026-07-26, so this buys diagnosability for the reviews that actually failed rather than growing all of them. The strings come from formatReviewDiagnosticsForCapture, whose `error` field is errorMessage() output and never raw provider text, so the public/private boundary is unchanged. Refs #9432 * fix(review): move the v8 ignore hint onto its own line so the type-level nullish arm is excluded codecov/patch flagged the `?? []` on the diagnostics spread as a partial: the ignore comment shared a line with the ternary's `?`, where the v8 provider does not register it. Match the working precedent at the PostHog capture site -- comment on its own line directly above the property -- and the branch drops out of the report. No behavior change.
1 parent 49b7ada commit 776c414

2 files changed

Lines changed: 64 additions & 3 deletions

File tree

src/queue/ai-review-orchestration.ts

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -987,23 +987,44 @@ export async function runAiReviewForAdvisory(
987987
}, "ai_review_inconclusive");
988988
}
989989
args.advisory.findings.push(...findings);
990+
// #9432: `degraded` persists the per-attempt review diagnostics into `ai_review_cache.metadata_json`.
991+
// Until now these existed ONLY as a PostHog capture property, which made the "why did this review produce
992+
// no summary?" question unanswerable from the database or the container logs -- the 2026-07-27
993+
// investigation had to stop there, because reaching PostHog needs an interactive OAuth the incident
994+
// responder may not have. The diagnostics are the discriminator that matters: `missing_assessment` (the
995+
// model genuinely returned no narrative) vs `unparseable_output`/`empty_output` (it returned something
996+
// parseModelReview could not read, i.e. no balanced JSON object) vs `provider_error` are three different
997+
// bugs with three different fixes, and `metadata_json.inconclusive` alone cannot tell them apart.
998+
//
999+
// Written on the DEGRADED paths only, never on the healthy one. Every byte here lands in a D1 row, and
1000+
// the 2026-07-26 outage was this database hitting its 10 GB ceiling -- so this buys diagnosability for
1001+
// the small minority of reviews that actually failed, rather than growing all of them. The compact
1002+
// `model#attempt:status[:error]` strings come from formatReviewDiagnosticsForCapture, whose `error` field
1003+
// is errorMessage() output and never raw provider text, so the public/private boundary is unchanged.
9901004
const metadataFor = (
9911005
notes: string | null | undefined,
9921006
inlineFindings: InlineFinding[],
1007+
degraded?: boolean,
9931008
): Record<string, unknown> => ({
9941009
rag: attributeReviewRagTelemetry(ragTelemetry, {
9951010
notes,
9961011
findings,
9971012
inlineFindings,
9981013
}),
1014+
...(degraded
1015+
? {
1016+
/* v8 ignore next -- current review runner always supplies diagnostics for completed AI attempts; the `?? []` is a type-level fallback for the optional field. */
1017+
reviewDiagnostics: formatReviewDiagnosticsForCapture(result.reviewDiagnostics ?? []),
1018+
}
1019+
: {}),
9991020
});
10001021
if (result.inconclusive && hasPublicReviewAssessment(result.advisoryNotes)) {
10011022
return {
10021023
notes: result.advisoryNotes!,
10031024
reviewerCount: result.reviewerCount,
10041025
inlineFindings: [],
10051026
findings,
1006-
metadata: metadataFor(result.advisoryNotes, []),
1027+
metadata: metadataFor(result.advisoryNotes, [], true),
10071028
cacheable: false,
10081029
valueAssessment: result.valueAssessment ?? undefined,
10091030
};
@@ -1026,7 +1047,7 @@ export async function runAiReviewForAdvisory(
10261047
reviewerCount: result.reviewerCount,
10271048
inlineFindings: [],
10281049
findings,
1029-
metadata: metadataFor(null, []),
1050+
metadata: metadataFor(null, [], true),
10301051
cacheable: false,
10311052
};
10321053
}
@@ -1068,7 +1089,7 @@ export async function runAiReviewForAdvisory(
10681089
reviewerCount: result.reviewerCount,
10691090
inlineFindings: [],
10701091
findings,
1071-
metadata: metadataFor(null, []),
1092+
metadata: metadataFor(null, [], true),
10721093
cacheable: false,
10731094
};
10741095
} catch (error) {

test/unit/ai-review-advisory.test.ts

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -814,6 +814,46 @@ describe("runAiReviewForAdvisory", () => {
814814
captureSpy.mockRestore();
815815
});
816816

817+
it("#9432: persists the review diagnostics into metadata when no public notes were produced", async () => {
818+
// The whole point: `metadata_json.inconclusive` says a review degraded but not WHY, and the diagnostics
819+
// used to reach PostHog only -- unreadable during an incident without an interactive OAuth. An empty
820+
// provider response must now be recoverable from the row itself as `empty_output`, which is what
821+
// distinguishes "the model returned nothing" from "the model returned a narrative we then dropped".
822+
const result = await runAiReviewForAdvisory(aiEnv(async () => ({ response: "" })), {
823+
mode: "live",
824+
settings: { aiReviewMode: "advisory" } as RepositorySettings,
825+
advisory: advisory(),
826+
repoFullName: "acme/widgets",
827+
pr,
828+
author: "alice",
829+
confirmedContributor: true,
830+
});
831+
const diagnostics = result?.metadata?.reviewDiagnostics as string[] | undefined;
832+
expect(Array.isArray(diagnostics)).toBe(true);
833+
expect(diagnostics?.length).toBeGreaterThan(0);
834+
// Compact `model#attempt:status` form from formatReviewDiagnosticsForCapture -- the status is the payload.
835+
expect(diagnostics?.every((entry) => /^[^#]+#\d+:/.test(entry))).toBe(true);
836+
expect(diagnostics?.join("\n")).toContain("empty_output");
837+
});
838+
839+
it("#9432: omits the review diagnostics from metadata on a healthy review", async () => {
840+
// The other side of the `degraded` branch. Diagnostics are deliberately NOT written on the success path:
841+
// every byte lands in a D1 row, and this database hit its 10 GB ceiling on 2026-07-26. Diagnosability is
842+
// bought only for the reviews that actually failed.
843+
const result = await runAiReviewForAdvisory(aiEnv(async () => ({ response: defectJson() })), {
844+
mode: "live",
845+
settings: { aiReviewMode: "advisory" } as RepositorySettings,
846+
advisory: advisory(),
847+
repoFullName: "acme/widgets",
848+
pr,
849+
author: "alice",
850+
confirmedContributor: true,
851+
});
852+
expect(result?.notes).toBeTruthy();
853+
expect(result?.metadata).toBeDefined();
854+
expect(result?.metadata?.reviewDiagnostics).toBeUndefined();
855+
});
856+
817857
it("#8790: stops retrying a model after a byte-identical repeat of a failed attempt (deterministic at temperature 0)", async () => {
818858
const adv = advisory();
819859
let aiCalls = 0;

0 commit comments

Comments
 (0)