feat(calibration): unify tightening under the backtest evidence standard (#8225) (#8267)

The registry gains direction-aware tightening: an optional per-knob ladder
declares its candidates, hard maximum, and the EXPLICIT axes orientation of
the trade (for the confidence-threshold corpus polarity, raising a bar helps
recall and risks precision — never the symmetric Pareto floor reused blind).
compareDirectionalBacktestScores enforces win-axis-strictly-up within a
declared sacrifice budget; evaluateKnobTightening/runKnobTightening transpose
the loosening discipline verbatim (same splits, same floors, smallest step
first); override storage validates per direction under per-direction flags
(tighten autonomy is its own default-off var); the drift sentinel's pool
gains declared ladders so tighter findings and the apply path judge the same
values. First ladder: ai_review_close_confidence 0.93 -> [0.95, 0.97].
Bespoke triggers (precision circuit-breakers) stay, with T3's constants
pinned byte-stable per the migration map on the issue.

Author: JSONbored

apps/loopover-ui/content/docs/backtest-calibration.mdx

@@ -108,6 +108,31 @@ loosening loop owns that direction). One alert per drift episode: a standing unc
 re-alerts, a changed one does, and recovery clears the episode. The sentinel is **alert-only** — it never
 writes a knob value — and the knobs status endpoint shows each knob's current drift report regardless of
 the flag, so a standing drift is always visible.
+## Tightening under the same evidence standard
+
+Historically the two directions ran on different evidence regimes: loosening moved only on Pareto-floored
+backtests, while tightening lived in a legacy loop (precision circuit-breakers plus a shadow-soaked
+advisor). The tightening unification brings threshold *raises* under the same backtest standard — with the
+axes orientation made **explicit** instead of reused blind. A deliberate tightening exists to move one axis
+at a bounded cost to the other; which axis is which depends on corpus polarity (for the confidence-threshold
+classifier, the positive class is "predicted reversed", so raising a bar helps *recall* and risks
+*precision* — the inverse of the rule-firing frame). Each knob's optional tightening ladder therefore
+declares its own orientation: the axis a raise must strictly improve, and the maximum sacrifice the other
+axis may suffer per comparison slice before the trade is a regression.
+
+Everything else is the loosening discipline transposed verbatim: the same split seed and fraction (held-out
+membership is identical in both directions), the same never-on-noise sample floors, smallest step first, a
+hard maximum no evidence may cross, and per-direction double gating — the tighten loop and the above-shipped
+override read are both gated by the ladder's **own** default-off flag, separate from the loosening flag, so
+each direction's autonomy is opted into independently and flipping either flag off instantly restores the
+shipped default for that direction. A declared ladder also joins the drift sentinel's candidate pool, so
+tighter-alternative findings and the tighten apply path judge the same values under the same floors.
+
+The legacy tightening triggers that are *not* threshold moves stay bespoke on purpose: the merge/close
+precision circuit-breakers are boolean capability breakers judged on realized outcomes (the strongest
+evidence regime there is), and re-basing them on fired-signal backtests would weaken, not strengthen, their
+grounds. Their behavior is pinned byte-stable by their own suites.
+
 ## Reliability curves
 
 Beside the candidate-ladder machinery, each live knob's status now carries its rule's

packages/loopover-engine/src/calibration/backtest-compare.ts

@@ -8,7 +8,7 @@
 import type { BacktestScoreReport } from "./backtest-score.js";
 
 /** The two comparable axes of a {@link BacktestScoreReport}. */
-type ComparisonAxis = "precision" | "recall";
+export type ComparisonAxis = "precision" | "recall";
 
 export type BacktestComparison = {
   ruleId: string;
@@ -50,3 +50,65 @@ export function compareBacktestScores(baseline: BacktestScoreReport, candidate:
     verdict: regressedAxes.length > 0 ? "regressed" : improvedAxes.length > 0 ? "improved" : "unchanged",
   };
 }
+
+/** The explicit axes orientation of a directional comparison (#8225): which axis the change exists to move
+ *  (and must move strictly up to earn "improved"), and how much the OTHER axis may be sacrificed for it. */
+export type DirectionalOrientation = {
+  mustImprove: ComparisonAxis;
+  /** Absolute drop the non-`mustImprove` axis may suffer before the trade is a regression. */
+  maxSacrifice: number;
+};
+
+/**
+ * Direction-aware comparator for a deliberate axis trade (#8225) -- a TIGHTENING exists to move one axis at
+ * a bounded cost to the other, so reusing the symmetric {@link compareBacktestScores} blind would brand
+ * every honest trade "regressed" the moment the sacrificed axis dips. Which axis is which depends on the
+ * corpus polarity (for the confidence-threshold classifier the positive class is "predicted reversed", so
+ * RAISING a threshold helps recall and risks precision -- the inverse of the rule-firing frame), hence the
+ * orientation is the CALLER's explicit declaration, never an assumption baked in here. The re-oriented
+ * floor:
+ *   • the `mustImprove` axis must move STRICTLY up for an "improved" verdict, and any drop on it is
+ *     "regressed" -- a trade that loses the axis it exists to win is simply wrong;
+ *   • the other axis may drop by at most `maxSacrifice`; a within-bound drop is the accepted trade and
+ *     appears in NEITHER axis list, an over-bound drop is "regressed", and a gain still counts;
+ *   • a null on either side of an axis excludes that axis entirely -- unknown stays unknown, exactly as in
+ *     the symmetric comparator (so a corpus with no judgeable win-axis can never yield "improved").
+ * Throws on a rule mismatch or a non-finite/negative bound: both are caller bugs, not valid comparisons.
+ */
+export function compareDirectionalBacktestScores(
+  baseline: BacktestScoreReport,
+  candidate: BacktestScoreReport,
+  orientation: DirectionalOrientation,
+): BacktestComparison {
+  if (baseline.ruleId !== candidate.ruleId) {
+    throw new Error(`cannot compare backtest scores for different rules: ${baseline.ruleId} vs ${candidate.ruleId}`);
+  }
+  if (!Number.isFinite(orientation.maxSacrifice) || orientation.maxSacrifice < 0) {
+    throw new Error(`maxSacrifice must be a non-negative finite number, got ${orientation.maxSacrifice}`);
+  }
+  const sacrificeAxis: ComparisonAxis = orientation.mustImprove === "precision" ? "recall" : "precision";
+  const regressedAxes: ComparisonAxis[] = [];
+  const improvedAxes: ComparisonAxis[] = [];
+  const winBaseline = baseline[orientation.mustImprove];
+  const winCandidate = candidate[orientation.mustImprove];
+  if (winBaseline !== null && winCandidate !== null) {
+    if (winCandidate < winBaseline) regressedAxes.push(orientation.mustImprove);
+    else if (winCandidate > winBaseline) improvedAxes.push(orientation.mustImprove);
+  }
+  const sacBaseline = baseline[sacrificeAxis];
+  const sacCandidate = candidate[sacrificeAxis];
+  if (sacBaseline !== null && sacCandidate !== null) {
+    if (sacBaseline - sacCandidate > orientation.maxSacrifice) regressedAxes.push(sacrificeAxis);
+    else if (sacCandidate > sacBaseline) improvedAxes.push(sacrificeAxis);
+  }
+  return {
+    ruleId: baseline.ruleId,
+    baseline,
+    candidate,
+    regressedAxes,
+    improvedAxes,
+    // "improved" requires the WIN axis specifically -- a lone gain on the sacrifice axis is not what the
+    // trade is for, so it stays "unchanged" (harmless, but no evidence the step earned its keep).
+    verdict: regressedAxes.length > 0 ? "regressed" : improvedAxes.includes(orientation.mustImprove) ? "improved" : "unchanged",
+  };
+}

packages/loopover-engine/test/backtest-compare.test.ts

@@ -1,7 +1,7 @@
 import assert from "node:assert/strict";
 import { test } from "node:test";
 
-import { compareBacktestScores, type BacktestScoreReport } from "../dist/index.js";
+import { compareBacktestScores, compareDirectionalBacktestScores, type BacktestScoreReport } from "../dist/index.js";
 
 function report(overrides: Partial<BacktestScoreReport> = {}): BacktestScoreReport {
   return {
@@ -61,3 +61,15 @@ test("compareBacktestScores: mismatched ruleIds throw, naming both rules", () =>
     /cannot compare backtest scores for different rules: missing_linked_issue vs other_rule/,
   );
 });
+
+test("barrel: the public entrypoint re-exports the direction-aware comparator (#8225)", () => {
+  assert.equal(typeof compareDirectionalBacktestScores, "function");
+});
+
+test("compareDirectionalBacktestScores: win axis up + within-budget sacrifice is improved; over-budget regresses", () => {
+  const orientation = { mustImprove: "recall" as const, maxSacrifice: 0.1 };
+  const ok = compareDirectionalBacktestScores(report(), report({ recall: 0.7, precision: 0.45 }), orientation);
+  assert.equal(ok.verdict, "improved");
+  const over = compareDirectionalBacktestScores(report(), report({ recall: 0.9, precision: 0.3 }), orientation);
+  assert.equal(over.verdict, "regressed");
+});

src/queue/job-dispatch.ts

@@ -31,7 +31,7 @@ import { isPrReconciliationEnabled, resolvePrReconciliationManifestOverride, run
 import { isActiveReviewReconciliationEnabled, resolveActiveReviewReconciliationManifestOverride, runActiveReviewReconciliation } from "../review/active-review-reconciliation";
 import { isSelfTuneEnabled, runSelfTune } from "../review/selftune-wire";
 import { isSatisfactionFloorAutotuneEnabled, runScheduledSatisfactionFloorLoosening } from "../services/satisfaction-floor-loosening-run";
-import { GENERIC_LIVE_KNOBS, isConfigDriftSentinelEnabled, isKnobAutotuneEnabled, runConfigDriftSentinel, runPerRepoKnobLoosening, runScheduledKnobLoosening } from "../services/knob-loosening-run";
+import { GENERIC_LIVE_KNOBS, isConfigDriftSentinelEnabled, isKnobAutotuneEnabled, isKnobTightenEnabled, runConfigDriftSentinel, runPerRepoKnobLoosening, runScheduledKnobLoosening, runScheduledKnobTightening } from "../services/knob-loosening-run";
 import { runSelfTuneBreaker } from "../review/outcomes-wire";
 import { isRagEnabled } from "../review/rag-wire";
 import { processSubmitDraft } from "../services/draft";
@@ -358,6 +358,9 @@ export async function processJob(env: Env, message: JobMessage): Promise<void> {
           // inherit global. Bounded per tick with a rotating cursor; fail-safe internally.
           await runPerRepoKnobLoosening(env, knob);
         }
+        // #8225: the tighten direction rides the same tick under its OWN per-knob default-off var —
+        // direction autonomy is opted into separately, never inherited from the loosening flag.
+        if (isKnobTightenEnabled(env, knob)) await runScheduledKnobTightening(env, knob);
       }
       // #8213: the drift sentinel rides the same calibration tick, behind its own default-off flag.
       // Alert-only — it never writes a knob value — and internally fail-safe per knob.