Skip to content

Commit d957ab8

Browse files
committed
feat(review): surface backtest-cleared loosening proposals in the tuning advisor (#8160)
computeTuningRecommendations still emits loosening advice as prose with no payload -- and the #8121 loop that can now actually measure a loosening was invisible to the advisor. buildSatisfactionFloorLooseningRecs (pure) turns the loop's state into ranked TuningRecs: a backtest-cleared proposal surfaces as a good-severity rec carrying both split verdicts, sample sizes, and the precision movement inline, with the action line matched to the flag state; a recently applied loosening surfaces as info pointing at the #8161 status surface. HARD BOUNDARY, tested: these recs never carry an overridePayload -- that field is the tightening-only auto-apply channel, so the advisor's apply path provably cannot promote a loosening. loadSatisfactionFloorRecState (evaluate-ONLY, fail-safe) feeds it from the same corpus + current floor the applying tick would use, and runSelfTune appends the recs exactly once per pass on the first repo's iteration (deployment-global state, never once per repo -- pinned by a spy test). 100% line+branch coverage on the new pure module; state-read paths (override- adjusted floor, empty corpus, broken DB, flag-off advice) all pinned. Closes #8160.
1 parent bd19df0 commit d957ab8

6 files changed

Lines changed: 240 additions & 2 deletions

File tree

src/review/loosening-recs.ts

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
// Loosening recommendations for the tuning advisor (#8160, sub-issue of epic #8121). auto-tune.ts's
2+
// computeTuningRecommendations deliberately emits loosening advice as prose with no payload (autonomous
3+
// loosening was the regression risk the loop existed to avoid — see OverridePayload's own doc). The #8121
4+
// narrow start made ONE loosening measurable (the satisfaction floor, backtest-gated); this module surfaces
5+
// that loop's state as TuningRec entries alongside the tightening recs, so the advisor's reader sees a
6+
// backtest-cleared loosening opportunity — or a recently-applied one — in the same ranked list.
7+
//
8+
// HARD BOUNDARY (the issue's own): these recs NEVER carry an `overridePayload`. That field is the
9+
// tightening-only auto-apply channel (runAutoApplyRecommendations consumes it); a loosening must only ever
10+
// be applied by the flag-gated loop itself (satisfaction-floor-loosening-run.ts), never promoted by the
11+
// advisor's apply path. PURE — no IO; the caller supplies the loop's state.
12+
import type { TuningRec } from "./auto-tune";
13+
import type { SatisfactionFloorLooseningProposal } from "../services/satisfaction-floor-loosening";
14+
15+
/** The advisor list is per-project elsewhere; the satisfaction floor is deployment-global, so its recs use
16+
* this fixed pseudo-project label rather than impersonating any repo. */
17+
export const LOOSENING_REC_PROJECT = "global:satisfaction-floor";
18+
19+
export type SatisfactionFloorRecInput = {
20+
flagEnabled: boolean;
21+
proposal: SatisfactionFloorLooseningProposal | null;
22+
/** created_at of the most recent applied loosening (calibration.satisfaction_floor_loosened), or null. */
23+
lastAppliedAt: string | null;
24+
};
25+
26+
const pct = (value: number | null): string => (value == null ? "—" : `${Math.round(value * 100)}%`);
27+
28+
/**
29+
* Build the loosening TuningRecs from the loop's current state. At most two entries, never a payload:
30+
* • a backtest-cleared PROPOSAL → severity `good` — the positive "evidence says this can loosen" signal,
31+
* with both split verdicts + sample sizes inline and the action that matches the flag state (flip the
32+
* flag vs. wait for the hourly tick);
33+
* • a recently APPLIED loosening → severity `info`, pointing at the operator status surface (#8161).
34+
* No state ⇒ []. Pure and deterministic.
35+
*/
36+
export function buildSatisfactionFloorLooseningRecs(input: SatisfactionFloorRecInput): TuningRec[] {
37+
const recs: TuningRec[] = [];
38+
if (input.proposal) {
39+
const { proposal } = input;
40+
const action = input.flagEnabled
41+
? "The autotune flag is ON — the hourly tick will apply this step automatically."
42+
: "The autotune flag is OFF — set SATISFACTION_FLOOR_AUTOTUNE_ENABLED (or POST /v1/internal/calibration/loosen-satisfaction-floor after flipping it) to let the loop act.";
43+
recs.push({
44+
project: LOOSENING_REC_PROJECT,
45+
severity: "good",
46+
message:
47+
`Backtest-cleared LOOSENING available: satisfaction confidence floor ${proposal.currentFloor}${proposal.proposedFloor}. ` +
48+
`Visible split ${proposal.visible.verdict} (${proposal.visibleCases} case(s), precision ${pct(proposal.visible.baseline.precision)}${pct(proposal.visible.candidate.precision)}); ` +
49+
`held-out split ${proposal.heldOut.verdict} (${proposal.heldOutCases} case(s)). ${action}`,
50+
// Deliberately NO overridePayload: that channel is tightening-only (see the module doc).
51+
});
52+
}
53+
if (input.lastAppliedAt) {
54+
recs.push({
55+
project: LOOSENING_REC_PROJECT,
56+
severity: "info",
57+
message: `A backtest-gated loosening was applied at ${input.lastAppliedAt} — see GET /v1/internal/calibration/satisfaction-floor for the live floor and full evidence history.`,
58+
});
59+
}
60+
return recs;
61+
}

src/review/selftune-wire.ts

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ import { buildRepoOutcomeCalibration } from "../services/outcome-calibration";
4141
import { loadRepoFocusManifest } from "../signals/focus-manifest-loader";
4242
import { errorMessage } from "../utils/json";
4343
import { computeTuningRecommendations, type GateEvalReport, type GateEvalRow } from "./auto-tune";
44+
import { buildSatisfactionFloorLooseningRecs } from "./loosening-recs";
45+
import { loadSatisfactionFloorRecState } from "../services/satisfaction-floor-loosening-run";
4446
import { runAutoApplyRecommendations, type StorageEnv } from "./auto-apply";
4547

4648
/** True when the self-improvement loop is enabled. Flag-OFF (default) → every export below is a no-op. Truthy
@@ -155,6 +157,11 @@ export async function runSelfTune(env: Env): Promise<void> {
155157
const row = await buildEvalRow(env, repoFullName);
156158
const report: GateEvalReport = { rows: [row], hasSignal: row.decided >= 10 };
157159
const recs = computeTuningRecommendations(report);
160+
// #8160: surface the backtest-gated loosening loop's state in the same ranked list. Payload-less by
161+
// design — runAutoApplyRecommendations below only ever consumes recs carrying a TIGHTENING
162+
// overridePayload, so these are report-only here and can never be promoted by the apply path.
163+
// Appended once per pass (deployment-global state), on the first repo's iteration.
164+
if (repoFullName === repos[0]) recs.push(...buildSatisfactionFloorLooseningRecs(await loadSatisfactionFloorRecState(env, nowMs)));
158165
// runAutoApplyRecommendations only ever consumes recs that carry a TIGHTENING overridePayload, shadow-
159166
// soaks them, and promotes a soaked override only when isStrictlyTightening + evidence + soak pass.
160167
await runAutoApplyRecommendations(env as unknown as StorageEnv, {

src/services/satisfaction-floor-loosening-run.ts

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,3 +133,42 @@ export async function runScheduledSatisfactionFloorLoosening(env: Env): Promise<
133133
return null;
134134
}
135135
}
136+
137+
/** The loop state the tuning advisor surfaces (#8160). Structurally what
138+
* src/review/loosening-recs.ts's builder consumes — kept here so the advisor wire needs one read. */
139+
export type SatisfactionFloorRecState = {
140+
flagEnabled: boolean;
141+
proposal: import("./satisfaction-floor-loosening").SatisfactionFloorLooseningProposal | null;
142+
lastAppliedAt: string | null;
143+
};
144+
145+
/**
146+
* Evaluate-ONLY state read for the advisor (#8160): the current proposal (from the same corpus + current
147+
* floor the applying tick would use — but never writing anything) plus the newest applied-loosening
148+
* timestamp. Fail-safe on every read: a corpus/history error degrades to a null section rather than
149+
* breaking the advisor surface that embeds this.
150+
*/
151+
export async function loadSatisfactionFloorRecState(env: Env, nowMs: number = Date.now()): Promise<SatisfactionFloorRecState> {
152+
const flagEnabled = isSatisfactionFloorAutotuneEnabled(env);
153+
154+
let proposal: SatisfactionFloorRecState["proposal"] = null;
155+
try {
156+
const currentFloor = (await getSatisfactionFloorOverride(env)) ?? LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR;
157+
const { fired, overrides } = await createSignalStore(env).queryRuleHistory(SATISFACTION_FLOOR_RULE_ID, nowMs - CORPUS_LOOKBACK_MS);
158+
proposal = evaluateSatisfactionFloorLoosening(buildBacktestCorpus(SATISFACTION_FLOOR_RULE_ID, fired, overrides), currentFloor);
159+
} catch {
160+
proposal = null;
161+
}
162+
163+
let lastAppliedAt: string | null = null;
164+
try {
165+
const row = await env.DB.prepare("SELECT created_at FROM audit_events WHERE event_type = ? ORDER BY created_at DESC LIMIT 1")
166+
.bind(SATISFACTION_FLOOR_LOOSENING_EVENT_TYPE)
167+
.first<{ created_at: string }>();
168+
lastAppliedAt = row?.created_at ?? null;
169+
} catch {
170+
lastAppliedAt = null;
171+
}
172+
173+
return { flagEnabled, proposal, lastAppliedAt };
174+
}

test/unit/loosening-recs.test.ts

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
import { describe, expect, it } from "vitest";
2+
import { buildSatisfactionFloorLooseningRecs, LOOSENING_REC_PROJECT } from "../../src/review/loosening-recs";
3+
import type { SatisfactionFloorLooseningProposal } from "../../src/services/satisfaction-floor-loosening";
4+
5+
function proposal(overrides: Partial<SatisfactionFloorLooseningProposal> = {}): SatisfactionFloorLooseningProposal {
6+
return {
7+
ruleId: "linked_issue_scope_mismatch",
8+
currentFloor: 0.5,
9+
proposedFloor: 0.45,
10+
visibleCases: 24,
11+
heldOutCases: 7,
12+
visible: {
13+
ruleId: "linked_issue_scope_mismatch",
14+
baseline: { ruleId: "linked_issue_scope_mismatch", caseCount: 24, truePositive: 1, falsePositive: 4, trueNegative: 19, falseNegative: 0, precision: 0.2, recall: 1 },
15+
candidate: { ruleId: "linked_issue_scope_mismatch", caseCount: 24, truePositive: 1, falsePositive: 0, trueNegative: 23, falseNegative: 0, precision: 1, recall: 1 },
16+
regressedAxes: [],
17+
improvedAxes: ["precision"],
18+
verdict: "improved",
19+
},
20+
heldOut: {
21+
ruleId: "linked_issue_scope_mismatch",
22+
baseline: { ruleId: "linked_issue_scope_mismatch", caseCount: 7, truePositive: 0, falsePositive: 0, trueNegative: 7, falseNegative: 0, precision: null, recall: null },
23+
candidate: { ruleId: "linked_issue_scope_mismatch", caseCount: 7, truePositive: 0, falsePositive: 0, trueNegative: 7, falseNegative: 0, precision: null, recall: null },
24+
regressedAxes: [],
25+
improvedAxes: [],
26+
verdict: "unchanged",
27+
},
28+
...overrides,
29+
};
30+
}
31+
32+
describe("buildSatisfactionFloorLooseningRecs (#8160)", () => {
33+
it("returns [] with no proposal and no applied history", () => {
34+
expect(buildSatisfactionFloorLooseningRecs({ flagEnabled: false, proposal: null, lastAppliedAt: null })).toEqual([]);
35+
});
36+
37+
it("surfaces a backtest-cleared proposal as a good-severity rec with both split verdicts, sample sizes, and precision movement", () => {
38+
const recs = buildSatisfactionFloorLooseningRecs({ flagEnabled: false, proposal: proposal(), lastAppliedAt: null });
39+
expect(recs).toHaveLength(1);
40+
const [rec] = recs;
41+
expect(rec!.project).toBe(LOOSENING_REC_PROJECT);
42+
expect(rec!.severity).toBe("good");
43+
expect(rec!.message).toContain("0.5 → 0.45");
44+
expect(rec!.message).toContain("Visible split improved (24 case(s), precision 20% → 100%)");
45+
expect(rec!.message).toContain("held-out split unchanged (7 case(s))");
46+
// Null precision renders as the em-dash convention, never a fake number.
47+
const nullPrecision = buildSatisfactionFloorLooseningRecs({
48+
flagEnabled: false,
49+
proposal: proposal({ visible: { ...proposal().visible, baseline: { ...proposal().visible.baseline, precision: null } } }),
50+
lastAppliedAt: null,
51+
});
52+
expect(nullPrecision[0]!.message).toContain("precision — →");
53+
});
54+
55+
it("the action line matches the flag state, and the rec NEVER carries an overridePayload (the tightening-only channel)", () => {
56+
const off = buildSatisfactionFloorLooseningRecs({ flagEnabled: false, proposal: proposal(), lastAppliedAt: null });
57+
expect(off[0]!.message).toContain("SATISFACTION_FLOOR_AUTOTUNE_ENABLED");
58+
const on = buildSatisfactionFloorLooseningRecs({ flagEnabled: true, proposal: proposal(), lastAppliedAt: null });
59+
expect(on[0]!.message).toContain("hourly tick will apply");
60+
for (const rec of [...off, ...on]) expect(rec.overridePayload).toBeUndefined();
61+
});
62+
63+
it("reports a recently applied loosening as an info rec pointing at the operator status surface, alongside a proposal when both exist", () => {
64+
const both = buildSatisfactionFloorLooseningRecs({ flagEnabled: true, proposal: proposal(), lastAppliedAt: "2026-07-23T05:00:00.000Z" });
65+
expect(both.map((rec) => rec.severity)).toEqual(["good", "info"]);
66+
expect(both[1]!.message).toContain("2026-07-23T05:00:00.000Z");
67+
expect(both[1]!.message).toContain("/v1/internal/calibration/satisfaction-floor");
68+
69+
const appliedOnly = buildSatisfactionFloorLooseningRecs({ flagEnabled: true, proposal: null, lastAppliedAt: "2026-07-23T05:00:00.000Z" });
70+
expect(appliedOnly).toHaveLength(1);
71+
expect(appliedOnly[0]!.severity).toBe("info");
72+
});
73+
});

test/unit/satisfaction-floor-loosening-run.test.ts

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import { splitBacktestCorpus } from "@loopover/engine";
33
import {
44
getSatisfactionFloorOverride,
55
isSatisfactionFloorAutotuneEnabled,
6+
loadSatisfactionFloorRecState,
67
runSatisfactionFloorLoosening,
78
runScheduledSatisfactionFloorLoosening,
89
SATISFACTION_FLOOR_LOOSENING_EVENT_TYPE,
@@ -225,3 +226,44 @@ describe("runScheduledSatisfactionFloorLoosening + queue wiring (#8158)", () =>
225226
expect(await getSatisfactionFloorOverride(onEnv)).toBe(core.SATISFACTION_FLOOR_LOOSENING_CANDIDATES[0]);
226227
});
227228
});
229+
230+
describe("loadSatisfactionFloorRecState (#8160)", () => {
231+
it("reports flag state, a live proposal from the current corpus, and null lastAppliedAt on a fresh deployment", async () => {
232+
const env = enabledEnv();
233+
await seedLooseningFriendlyHistory(env);
234+
const state = await loadSatisfactionFloorRecState(env);
235+
expect(state.flagEnabled).toBe(true);
236+
expect(state.proposal?.proposedFloor).toBe(core.SATISFACTION_FLOOR_LOOSENING_CANDIDATES[0]);
237+
expect(state.lastAppliedAt).toBeNull();
238+
});
239+
240+
it("evaluates from the OVERRIDDEN floor when one is live, and reads the newest applied timestamp", async () => {
241+
const env = enabledEnv();
242+
await seedLooseningFriendlyHistory(env);
243+
// Apply once for real: the next state read must evaluate from 0.45 (no further proposal on this corpus)
244+
// and report the applied event's timestamp.
245+
expect((await runSatisfactionFloorLoosening(env)).applied).toBe(true);
246+
const state = await loadSatisfactionFloorRecState(env);
247+
expect(state.proposal).toBeNull();
248+
expect(state.lastAppliedAt).not.toBeNull();
249+
});
250+
251+
it("stays proposal-null on an empty corpus and fails safe to nulls on a broken DB", async () => {
252+
const fresh = await loadSatisfactionFloorRecState(enabledEnv());
253+
expect(fresh.proposal).toBeNull();
254+
expect(fresh.lastAppliedAt).toBeNull();
255+
256+
const env = enabledEnv();
257+
env.DB = { prepare: () => { throw new Error("boom"); } } as never;
258+
const broken = await loadSatisfactionFloorRecState(env);
259+
expect(broken).toEqual({ flagEnabled: true, proposal: null, lastAppliedAt: null });
260+
});
261+
262+
it("flag off: reports flagEnabled false while still evaluating from the shipped floor (advice can precede the opt-in)", async () => {
263+
const env = createTestEnv();
264+
await seedLooseningFriendlyHistory(env);
265+
const state = await loadSatisfactionFloorRecState(env);
266+
expect(state.flagEnabled).toBe(false);
267+
expect(state.proposal?.currentFloor).toBe(0.5);
268+
});
269+
});

test/unit/selftune-wiring.test.ts

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ async function seedRegisteredRepo(env: Env, fullName: string, autonomyJson: stri
4545
// Seed N resolved recommendation outcomes for a repo: `negative` rejected/closed (the dangerous error a
4646
// tightening fixes) + `positive` accepted. Inserted directly (FKs off) — buildRepoOutcomeCalibration reads
4747
// the outcome_state split, which is all the eval mapping needs.
48-
async function seedRecommendationOutcomes(env: Env, repoFullName: string, positive: number, negative: number, maintainerLane = true): Promise<void> {
48+
async function seedRecommendationOutcomes(env: Env, repoFullName: string, positive: number, negative: number, maintainerLane = true, idPrefix = ""): Promise<void> {
4949
await env.DB.prepare("PRAGMA foreign_keys=OFF").run();
5050
let i = 0;
5151
const insert = async (state: string) => {
@@ -56,7 +56,7 @@ async function seedRecommendationOutcomes(env: Env, repoFullName: string, positi
5656
maintainer_lane, confidence, reason, source, updated_at)
5757
VALUES (?, ?, ?, ?, 'review', ?, 'pull_request', ?, ?, 'medium', 'seed', 'inferred', CURRENT_TIMESTAMP)`,
5858
)
59-
.bind(`o${i}`, `a${i}`, `r${i}`, "bot", state, repoFullName, maintainerLane ? 1 : 0)
59+
.bind(`${idPrefix}o${i}`, `${idPrefix}a${i}`, `${idPrefix}r${i}`, "bot", state, repoFullName, maintainerLane ? 1 : 0)
6060
.run();
6161
};
6262
for (let n = 0; n < positive; n += 1) await insert("accepted");
@@ -234,6 +234,22 @@ describe("runSelfTune — shadow-soak over loopover's own outcome data", () => {
234234
expect((await listOverrideAudit(env as never, "owner/repo")).length).toBe(0);
235235
});
236236

237+
it("appends the loosening-loop recs exactly once per pass, on the first repo only (#8160)", async () => {
238+
const state = await import("../../src/services/satisfaction-floor-loosening-run");
239+
const spy = vi.spyOn(state, "loadSatisfactionFloorRecState");
240+
const env = createTestEnv({ LOOPOVER_REVIEW_SELFTUNE: "true" });
241+
await seedRegisteredRepo(env, "owner/repo", ACTING_AUTONOMY);
242+
await seedRegisteredRepo(env, "owner/other", ACTING_AUTONOMY);
243+
await seedRecommendationOutcomes(env, "owner/repo", 5, 10);
244+
await seedRecommendationOutcomes(env, "owner/other", 5, 10, true, "b-");
245+
246+
await processJob(env, { type: "selftune", requestedBy: "schedule" });
247+
248+
// Two repos in the pass, ONE deployment-global loosening-state read: the recs are appended on the
249+
// first repo's iteration only, never once per repo.
250+
expect(spy).toHaveBeenCalledTimes(1);
251+
});
252+
237253
it("FLAG-ON via the processor: a stale in-flight selftune job runs the tick (defense-in-depth gate)", async () => {
238254
const env = createTestEnv({ LOOPOVER_REVIEW_SELFTUNE: "true" });
239255
await seedRegisteredRepo(env, "owner/repo", ACTING_AUTONOMY);

0 commit comments

Comments
 (0)