Proposed protected change
Follow-up to #2289. The repeatability test currently uses substring assertions such as "STABLE over 2 runs" in headline; because STABLE over N runs is contained in UNSTABLE over N runs, a regression to the negative verdict can pass.
Trust impact
This is a tests-only correction to the benchmark-integrity checks. It makes the test reject a false stable result; it does not change benchmark scoring or production behavior.
Scope
- anchor stable and unstable headline assertions at the
repeatability: ... prefix
- add one regression case proving an
UNSTABLE headline cannot satisfy the stable assertion
- leave implementation and benchmark outputs unchanged
Validation
- focused repeatability tests: 39 passed
- Ruff on
tests/test_repeatability.py: passed
git diff --check: passed
Prepared with OpenAI Codex under human direction, with the change reviewed before submission. Please apply benchmark-change-approved if this protected tests-only change is acceptable, or let me know what scope should change.
Proposed protected change
Follow-up to #2289. The repeatability test currently uses substring assertions such as
"STABLE over 2 runs" in headline; becauseSTABLE over N runsis contained inUNSTABLE over N runs, a regression to the negative verdict can pass.Trust impact
This is a tests-only correction to the benchmark-integrity checks. It makes the test reject a false stable result; it does not change benchmark scoring or production behavior.
Scope
repeatability: ...prefixUNSTABLEheadline cannot satisfy the stable assertionValidation
tests/test_repeatability.py: passedgit diff --check: passedPrepared with OpenAI Codex under human direction, with the change reviewed before submission. Please apply
benchmark-change-approvedif this protected tests-only change is acceptable, or let me know what scope should change.