Skip to content

Commit 102c952

Browse files
authored
Fix(eval): sync test-ignore updates from internal reference (#40)
Port the June 2026 test-ignore updates from the internal reference harness into ProgramBench across 41 instances (+270 ignored tests total), keeping all metadata verbatim, and update CLAUDE.md accordingly. Branch-hash keys and per-branch test name-sets are unchanged, so no test blob re-upload is needed; only ignore metadata differs. The testorg__calculator fixture is untouched. Internal-reference-commit: 1ca7719a46b9fc521af2353e287b4f07f4c071ea Internal-reference-commit: a8f153ad485daf95e2041b5e872b5b715b7f5410 Internal-reference-commit: b6466b9e33c2abd88ce8ca06037c876b1edcc864 Internal-reference-commit: d7f13854873fd23b60606729d649f31a4dc7d7ac Internal-reference-commit: 7b3a00e057cc625c86bcb9988a439ae50adb0828 Internal-reference-commit: 83488589c5237b4b9af904e1809997df735106f0 Internal-reference-commit: 3bbf4632e7d5e978b817bc168fa0897d0bbc2161 Internal-reference-commit: 691dcaa75a89d8d6f22fc7253c641375df9c3bdf
1 parent 57a6d3c commit 102c952

42 files changed

Lines changed: 3714 additions & 2 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CLAUDE.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,26 @@
44

55
ProgramBench evaluates whether LM-based SWE-agents can reverse-engineer black-box software systems. The workflow: take an open-source CLI tool (mostly Rust/Go), compile it into a Docker image with source removed, then have an LM agent re-implement it from scratch by interacting only with the binary. Behavioral tests (also LM-generated) score the re-implementation.
66

7+
## Test ignore reasons
8+
9+
Some behavioral tests are unreliable and are excluded from scoring. Each excluded
10+
test is recorded under `branches.<hash>.ignored_tests[]` in a task's `tests.json`,
11+
with one or more `reasons[].id` explaining why. All ignored tests are excluded from
12+
scoring regardless of reason; the id is informational.
13+
14+
- `gold_fail` — test fails **deterministically** on the reference (gold) solution, so it
15+
is defective rather than discriminating. Also covers golden-output drift (the gold
16+
binary is correct but the captured golden is stale/non-reproducible relative to the
17+
build toolchain, an embedded build-stamp, or an external resource).
18+
- `gold_flaky` — test is **non-deterministic** on the gold solution: it passes in some
19+
runs and fails in others. These are timing/race/network/TUI-snapshot flakes, not real
20+
defects (distinct from the deterministic `gold_fail`).
21+
- `dummy_pass` — test passes even on a trivial/dummy executable, so it fails to
22+
distinguish a real implementation from a stub.
23+
- `outcome_dependent_presence` — test appears in some eval runs but not others.
24+
- `slow_or_hang` — test hangs mid-call or exceeds a duration threshold.
25+
- `ignored_manual` — manually excluded.
26+
727
## Quick reference
828

929
```bash

0 commit comments

Comments
 (0)