Skip to content

Protected-change approval: harden repeatability headline assertions #2370

Description

@tomatotomata

Proposed protected change

Follow-up to #2289. The repeatability test currently uses substring assertions such as "STABLE over 2 runs" in headline; because STABLE over N runs is contained in UNSTABLE over N runs, a regression to the negative verdict can pass.

Trust impact

This is a tests-only correction to the benchmark-integrity checks. It makes the test reject a false stable result; it does not change benchmark scoring or production behavior.

Scope

  • anchor stable and unstable headline assertions at the repeatability: ... prefix
  • add one regression case proving an UNSTABLE headline cannot satisfy the stable assertion
  • leave implementation and benchmark outputs unchanged

Validation

  • focused repeatability tests: 39 passed
  • Ruff on tests/test_repeatability.py: passed
  • git diff --check: passed

Prepared with OpenAI Codex under human direction, with the change reviewed before submission. Please apply benchmark-change-approved if this protected tests-only change is acceptable, or let me know what scope should change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    benchmarkEvaluation harnesstestsTest suite

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions