feat(calibration): counterfactual replay contract — fixtures, spend, and scoring decided - #8242
Merged
Merged
Conversation
…and scoring decided (#8219) The design deliverable the counterfactual sub-epic (#8218) implements against, as typed constants/interfaces the assembler (#8220) and harness (#8221) must cite rather than reinvent: - Fixtures: replayable = bounded diff + human label; capture-era provenance carried so era-skewed improvements are visible; over-budget selection is a SEEDED deterministic sample recorded per run — never 'the first N'. - Spend: neuron-estimate units matching live AI budgeting, hard per-run cap defaulting well under one day's review spend, partial-run persist + resume cursor (the #8170 discipline), local-model smoke path, BYOK behind explicit flags, offline-only. - Scoring: a variant is just another classifier — scoreBacktest and the Pareto comparator apply unchanged; unparseable output is an ABSTENTION, counted and excluded, never coerced (coercion would let a degenerate prompt farm precision by failing on hard cases). Docs page gains the design section; contract module 100% covered by its mirror suite. Closes #8219
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 7e3a074 | Commit Preview URL Branch Preview URL |
Jul 23 2026, 02:09 PM |
Bundle ReportChanges will increase total bundle size by 3.97kB (0.05%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8242 +/- ##
==========================================
- Coverage 92.11% 92.11% -0.01%
==========================================
Files 780 781 +1
Lines 78489 78492 +3
Branches 23712 23713 +1
==========================================
- Hits 72304 72303 -1
Misses 5062 5062
- Partials 1123 1127 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
loopover-orb
Bot
added
the
gittensor:bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The #8219 design deliverable for the counterfactual sub-epic (#8218): the three contracts the assembler (#8220) and harness (#8221) implement against, shipped as typed interfaces + constants in the engine plus the docs design section — decisions in code, not prose.
scoreBacktest/compareBacktestScoresapply unchanged; unparseable output is an ABSTENTION, counted and excluded, never coerced (coercion would let a degenerate prompt farm precision by failing on hard cases).Validation
Full
npm run test:cigreen (TEST_CI_EXIT=0); the contract module is 100% covered by its root mirror suite (every arm of the shared replayability rule + pinned constants).Closes #8219