Skip to content

Commit 3539934

Browse files
authored
feat(calibration): consume the ai_review_close_confidence override and flip the knob to live (#8197)
The #8176 gate-authority change, mirroring the satisfaction floor's consumption discipline exactly: - One bounds-validated resolution point: getKnobOverride reads the knob's system_flags row only when its own wrangler var (AI_REVIEW_CLOSE_CONFIDENCE_AUTOTUNE_ENABLED, false by default) is ON, and refuses anything not strictly below the shipped 0.93 or under the 0.85 hard minimum. gateCheckPolicy threads it as the LAST-resort default — an explicit per-repo gate.aiReview.closeConfidence setting always wins. - Generic apply machinery (knob-loosening-run.ts): the satisfaction run/status/tick generalized over the registry entry rather than duplicated. The satisfaction floor keeps its legacy module and event shape; the shared cron tick now runs every OTHER live knob, double-gated per knob. Report-only refusal stays enforced in the write path itself. - Registry entries carry their apply plumbing (override flag key, event type, autotune var), pinned against the legacy constants by invariant tests; the close-confidence knob flips to applyMode live. - Operator surface: GET /v1/internal/calibration/knobs renders every live knob's flag state, shipped/live/override values, and applied history through one projector that reads both proposal spellings. Closes #8176
1 parent f5993e8 commit 3539934

14 files changed

Lines changed: 637 additions & 19 deletions

src/api/routes.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,7 @@ import { getPublicStats, isPublicStatsEnabled, resolvePublicStatsManifestOverrid
321321
import { loadPublicAccuracyTrend } from "../services/public-accuracy-trend";
322322
import { loadCalibrationTrend } from "../services/rule-calibration-trend";
323323
import { isSatisfactionFloorAutotuneEnabled, loadSatisfactionFloorStatus, runSatisfactionFloorLoosening } from "../services/satisfaction-floor-loosening-run";
324+
import { loadLiveKnobStatuses } from "../services/knob-loosening-run";
324325
import { loadPublicReuseRateTrend } from "../services/public-reuse-rate-trend";
325326
import { loadPublicReviewVolumeTrend } from "../services/public-review-volume-trend";
326327
import { buildMaintainerQualityDashboard, isMaintainerQualityDataStale } from "../services/maintainer-quality-dashboard";
@@ -4832,6 +4833,11 @@ export function createApp() {
48324833
// off. Same INTERNAL_JOB_TOKEN gate via the /v1/internal/* middleware; aggregate numbers/verdicts only.
48334834
app.get("/v1/internal/calibration/satisfaction-floor", async (c) => c.json(await loadSatisfactionFloorStatus(c.env)));
48344835

4836+
// The #8161 surface generalized across EVERY live registry knob (#8176): one endpoint, one projector,
4837+
// per-knob flag state + shipped/live/override values + applied history (both split verdicts). Same
4838+
// deliberate non-flag-gating and INTERNAL_JOB_TOKEN posture as the satisfaction-floor read above.
4839+
app.get("/v1/internal/calibration/knobs", async (c) => c.json({ knobs: await loadLiveKnobStatuses(c.env) }));
4840+
48354841
app.post("/v1/internal/jobs/refresh-registry", async (c) => {
48364842
const message: JobMessage = { type: "refresh-registry", requestedBy: "api" };
48374843
await c.env.JOBS.send(message);

src/queue/gate-checks.ts

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,10 @@ export function gateCheckPolicy(
6666
guardrailHit: boolean;
6767
guardrailMatches?: ReturnType<typeof guardrailPathMatches> | undefined;
6868
},
69+
// #8176: the backtest-gated GLOBAL default-override for the AI close-confidence floor, resolved by the
70+
// env-bearing caller (getAiReviewCloseConfidenceOverride — flag-gated + bounds-validated). It only fills
71+
// the DEFAULT: an explicit per-repo `gate.aiReview.closeConfidence` setting always wins below.
72+
aiReviewCloseConfidenceOverride?: number | null,
6973
) {
7074
// `settings` is already the EFFECTIVE config (`.loopover.yml` > DB > defaults), resolved upstream by
7175
// resolveRepositorySettings, so the blocker modes here reflect the repo's config file directly.
@@ -81,8 +85,9 @@ export function gateCheckPolicy(
8185
qualityGateMinScore: settings.qualityGateMinScore ?? null,
8286
aiReviewGateMode: settings.aiReviewMode,
8387
// Calibrated AI close-confidence floor (#7) — config-as-code via `.loopover.yml gate.aiReview.closeConfidence`,
84-
// resolved into settings upstream. `null`/undefined ⇒ advisory.ts applies the 0.93 default.
85-
aiReviewCloseConfidence: settings.aiReviewCloseConfidence ?? null,
88+
// resolved into settings upstream. When the repo has no explicit setting, the #8176 backtest-gated
89+
// global override (if any) becomes the default; `null` ⇒ advisory.ts applies the 0.93 shipped default.
90+
aiReviewCloseConfidence: settings.aiReviewCloseConfidence ?? aiReviewCloseConfidenceOverride ?? null,
8691
// Sub-floor AI-judgment disposition (#4603) — DB-backed (dashboard-settable) + `.loopover.yml
8792
// gate.aiReview.lowConfidenceDisposition` override, resolved into settings upstream. `null`/undefined ⇒
8893
// advisory.ts applies the "hold_for_review" default.

src/queue/job-dispatch.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ import { isPrReconciliationEnabled, resolvePrReconciliationManifestOverride, run
3131
import { isActiveReviewReconciliationEnabled, resolveActiveReviewReconciliationManifestOverride, runActiveReviewReconciliation } from "../review/active-review-reconciliation";
3232
import { isSelfTuneEnabled, runSelfTune } from "../review/selftune-wire";
3333
import { isSatisfactionFloorAutotuneEnabled, runScheduledSatisfactionFloorLoosening } from "../services/satisfaction-floor-loosening-run";
34+
import { GENERIC_LIVE_KNOBS, isKnobAutotuneEnabled, runScheduledKnobLoosening } from "../services/knob-loosening-run";
3435
import { runSelfTuneBreaker } from "../review/outcomes-wire";
3536
import { isRagEnabled } from "../review/rag-wire";
3637
import { processSubmitDraft } from "../services/draft";
@@ -348,6 +349,11 @@ export async function processJob(env: Env, message: JobMessage): Promise<void> {
348349
// flag is ON, but a stale in-flight job landing after a flag-flip must still no-op. Never throws into
349350
// the queue (the scheduled wrapper fails safe).
350351
if (isSatisfactionFloorAutotuneEnabled(env)) await runScheduledSatisfactionFloorLoosening(env);
352+
// #8176: every LATER live registry knob rides the same tick through the generic runner — each knob
353+
// is double-gated on its OWN wrangler var, so an un-flagged knob does zero work here.
354+
for (const knob of GENERIC_LIVE_KNOBS) {
355+
if (isKnobAutotuneEnabled(env, knob)) await runScheduledKnobLoosening(env, knob);
356+
}
351357
return;
352358
case "selftune":
353359
// Convergence (self-improve / auto-tune, flag LOOPOVER_REVIEW_SELFTUNE). Defense-in-depth: the cron only

src/queue/processors.ts

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -426,6 +426,7 @@ export { runRetentionPrune } from "./retention";
426426
// test/unit/gate-check-policy.test.ts and test/unit/repository-settings-enforcement.test.ts's existing
427427
// `import { gateCheckPolicy } from "../../src/queue/processors"` keeps working unchanged.
428428
import { auditGateCheckPermissionMissing, gateCheckPolicy, recordPublishedGateCheckSummary } from "./gate-checks";
429+
import { getAiReviewCloseConfidenceOverride } from "../services/knob-loosening-run";
429430
export { gateCheckPolicy } from "./gate-checks";
430431
// #4013 step 9: same shim shape for the AI-review-orchestration functions -- imported here for this file's
431432
// own remaining internal callers, and re-exported so the many existing tests importing claimAiReviewLock,
@@ -1538,6 +1539,9 @@ export async function sweepRepoRegate(
15381539
// isScheduledRegateSweepJob (queue-common.ts) misclassifies it as background maintenance and it inherits the
15391540
// exact starvation this priority mechanism exists to avoid. Ordinary stale candidates keep the sweep prefix
15401541
// unchanged.
1542+
// #8176: the global close-confidence default-override, resolved once for the sweep (same value the main
1543+
// webhook path threads; an explicit per-repo setting still wins inside gateCheckPolicy).
1544+
const sweepCloseConfidenceOverride = await getAiReviewCloseConfidenceOverride(env);
15411545
for (const [index, pr] of candidates.entries()) {
15421546
const others = openPullRequests.filter(
15431547
(other) => other.number !== pr.number,
@@ -1561,7 +1565,7 @@ export async function sweepRepoRegate(
15611565
});
15621566
const gate = evaluateGateCheck(
15631567
advisory,
1564-
gateCheckPolicy(settings, null, undefined, pr.slopRisk ?? null),
1568+
gateCheckPolicy(settings, null, undefined, pr.slopRisk ?? null, undefined, undefined, sweepCloseConfidenceOverride),
15651569
);
15661570
verdicts[String(pr.number)] = gate.conclusion;
15671571
if (gate.conclusion === "failure" || gate.conclusion === "action_required")
@@ -10381,6 +10385,8 @@ async function maybePublishPrPublicSurface(
1038110385
slopRisk,
1038210386
authorHistory,
1038310387
gateSizeContext,
10388+
// #8176: backtest-gated global default for the close-confidence floor (explicit per-repo wins inside).
10389+
await getAiReviewCloseConfidenceOverride(env),
1038410390
);
1038510391
gateEvaluation = await withReviewPipelineSpan(
1038610392
"selfhost.review.gate",
@@ -11871,7 +11877,7 @@ async function maybeProcessResolveCommand(env: Env, deliveryId: string, payload:
1187111877
if (!findingRef.ok) { await recordAuditEvent(env, { eventType: "github_app.finding_resolved_skipped", actor: req.actor, targetKey, outcome: "completed", detail: findingRef.reason, metadata: { deliveryId, repoFullName: req.repoFullName, reason: findingRef.reason } }); await recordGithubProductUsage(env, "finding_resolved_skipped", { actor: req.actor, repoFullName: req.repoFullName, targetKey, outcome: "skipped", metadata: { reason: findingRef.reason } }); return true; }
1187211878
const { advisory } = await buildAuthorizedPrActionAdvisory(env, req.repoFullName, pr, settings);
1187311879
await appendPublishedAiReviewFindingsForResolve(env, req.repoFullName, pr, settings.aiReviewMode, advisory);
11874-
const gate = evaluateGateCheck(advisory, gateCheckPolicy(settings, null, undefined, pr.slopRisk ?? null));
11880+
const gate = evaluateGateCheck(advisory, gateCheckPolicy(settings, null, undefined, pr.slopRisk ?? null, undefined, undefined, await getAiReviewCloseConfidenceOverride(env)));
1187511881
const selection = selectWarningsForResolve(gate.warnings, findingRef);
1187611882
if (selection.reason === "finding_not_found") { await recordAuditEvent(env, { eventType: "github_app.finding_resolved_skipped", actor: req.actor, targetKey, outcome: "completed", detail: selection.reason, metadata: { deliveryId, repoFullName: req.repoFullName, reason: selection.reason } }); await recordGithubProductUsage(env, "finding_resolved_skipped", { actor: req.actor, repoFullName: req.repoFullName, targetKey, outcome: "skipped", metadata: { reason: selection.reason } }); return true; }
1187711883
const mode = resolveAgentActionMode({ globalPaused: isGlobalAgentPause(env) || (await isGlobalAgentFrozen(env)), agentPaused: settings.agentPaused, agentDryRun: settings.agentDryRun });
@@ -12102,7 +12108,7 @@ async function maybeProcessExplainCommand(env: Env, deliveryId: string, payload:
1210212108
}
1210312109
const { advisory } = await buildAuthorizedPrActionAdvisory(env, req.repoFullName, pr, settings);
1210412110
await appendPublishedAiReviewFindingsForResolve(env, req.repoFullName, pr, settings.aiReviewMode, advisory);
12105-
const gate = evaluateGateCheck(advisory, gateCheckPolicy(settings, null, undefined, pr.slopRisk ?? null));
12111+
const gate = evaluateGateCheck(advisory, gateCheckPolicy(settings, null, undefined, pr.slopRisk ?? null, undefined, undefined, await getAiReviewCloseConfidenceOverride(env)));
1210612112
const selection = selectWarningsForResolve(gate.warnings, findingRef);
1210712113
if (selection.reason === "finding_not_found") {
1210812114
const notFound = sanitizePublicComment([AGENT_COMMAND_COMMENT_MARKER, "", "> [!NOTE]", `> **No review finding \`${findingRef.findingCode}\` on this PR**`, "> That id is not among this PR's current review findings — re-run `@loopover explain <finding-id>` with an id from the review summary.", "", "---", loopoverFooter(env)].join("\n"));

0 commit comments

Comments
 (0)