14 infrastructure bugs found in SkillsBench Trial 1 audit — verifier, permissions, provisioning

[xdotli]

Summary

8-agent trajectory audit of SkillsBench Opus 4.7 with-skills Trial 1 (84 tasks) found 14 tasks with infrastructure bugs that caused false failures or prevented agent execution. These bugs reduce the measured score from a potential ~50-55% to 40.3%.

Model: claude-opus-4-7 | Agent: claude-agent-acp | Environment: Daytona | Concurrency: 2

Trial job dir: experiments/skillsbench/jobs/opus47-with-skills-t1/2026-04-24__16-00-14/

---

Category 1: Verifier Broken (5 tasks)

Agent did correct work but verifier failed to evaluate it. These are FALSE FAILs — reward=0.0 when it should be 1.0.

1. suricata-custom-exfil

• Bug: Verifier pip install pytest installs to a different Python than /usr/bin/python3. Verifier runs python3 -m pytest → No module named pytest.
• Agent work: Wrote correct Suricata rule, tested it, sid:1000001 fires on positive pcap.
• Trial dir: suricata-custom-exfil__4473b0c0 (retry that ran)
• Reproduce: Run the task, check verifier log for No module named pytest
• Fix: Pin Python path in verifier test.sh, or use uv run pytest instead of python3 -m pytest

2. video-tutorial-indexer

• Bug: Verifier installs pytest-json-ctrf==0.1.0 but the plugin registers differently — pytest -p ctrf fails with No module named 'ctrf'. Zero tests ran.
• Agent work: Transcribed video with Whisper, wrote tutorial_index.json with 29 chapters.
• Trial dir: video-tutorial-indexer__4d31a011
• Fix: Update ctrf plugin import name in test.sh or remove -p ctrf flag

3. sales-pivot-analysis

• Bug: Verifier test.sh only installs pytest + pytest-json-ctrf, but test_outputs.py imports openpyxl at line 9. ModuleNotFoundError. Zero tests ran.
• Agent work: Parsed PDF + XLSX, wrote demographic_analysis.xlsx with pivot tables.
• Trial dir: sales-pivot-analysis__f7896973
• Fix: Add openpyxl to verifier test.sh pip install

4. pddl-tpp-planning

• Bug: Verifier validate_plan() opens task01.pkl (reference solution pickle) which was never provisioned into /tests. FileNotFoundError.
• Agent work: Read PDDL domain/problem files, wrote valid plan files.
• Trial dir: pddl-tpp-planning__0ad04ccc
• Fix: Include reference .pkl files in task tests/ directory

5. fix-build-google-auto

• Bug: Verifier test.sh calls uv which is not installed in the sandbox. Dockerfile comment says "uv is installed" but never installs it. All 5 test lines fail with uv: command not found.
• Agent work: Identified 4 cascading JDK 11 build issues, wrote 8 patches, got mvn verify -DskipTests=true to BUILD SUCCESS.
• Trial dir: fix-build-google-auto__dc1de096
• Fix: Add uv install to Dockerfile

---

Category 2: Sandbox Permissions (4 tasks)

Agent did correct work but couldn't write to required output paths. sandbox_user=agent can't write to root-owned directories.

6. syzkaller-ppdev-syzlang

• Bug: /opt/syzkaller/sys/linux/ is owned by root. Agent (uid=1001) gets EACCES writing dev_ppdev.txt. Tried sudo (not installed), chmod (not permitted).
• Agent work: Correctly identified ppdev ioctl interface, computed ioctl numbers, wrote proper syzlang description.
• Trial dir: syzkaller-ppdev-syzlang__d928daed
• Fix: chmod -R 777 /opt/syzkaller/sys/linux/ in Dockerfile or add agent to appropriate group

7. taxonomy-tree-merge

• Bug: (a) Environment setup took 369s (vs typical 30-100s), eating 41% of 900s timeout. (b) /root/output/ owned by root, agent can't write.
• Agent work: Processed 8,718 records through embedding + clustering pipeline. Completed successfully but hit PermissionError on output.
• Trial dir: taxonomy-tree-merge__ce812f15
• Fix: Create output dir with agent permissions in Dockerfile; investigate slow build

8. lean4-proof

• Bug: Lean toolchain at /root/.elan/bin/ not accessible to sandbox_user=agent. Agent can't run lake or lean to type-check proofs.
• Agent work: Wrote a Lean 4 proof (blind, couldn't verify).
• Trial dir: lean4-proof__336be05e
• Fix: Install Lean toolchain in a shared path or grant agent access to /root/.elan/

9. multilingual-video-dubbing

• Bug: (a) /outputs directory root-owned, agent can't write. (b) Verifier itself crashes: No module named 'torch' — torch not installed for verifier.
• Agent work: Generated TTS via Gemini API, did audio processing.
• Trial dir: multilingual-video-dubbing__eb0c860f
• Fix: chmod /outputs in Dockerfile; add torch to verifier deps

---

Category 3: Environment Provisioning (2 tasks)

Input data missing from sandbox — agent has nothing to work with.

10. organize-messy-files

• Bug: Sandbox only had 3 files (DAMOP.pptx, paper_file_1.docx, paper_file_2.docx). Verifier expects ~100 PDFs across 5+ subject folders.
• Trial dir: organize-messy-files__7283c7f8
• Fix: Ensure all PDFs are included in the Docker build context / COPY step

11. azure-bgp-oscillation-route-leak

• Bug: Dockerfile's inline DATAGEN Python script should generate JSON files into /app/data/. In sandbox, /app/data/ was empty.
• Trial dir: azure-bgp-oscillation-route-leak__d4273147
• Fix: Debug the DATAGEN heredoc in Dockerfile — may fail silently during Daytona image build

---

Category 4: Agent Launch / SDK Failures (3 tasks)

12. fix-build-agentops

• Bug: Agent binary claude-agent-acp not found (rc=127). bugswarm/cached-images base image has PATH issues for Node.js/npm.
• Trial dir: fix-build-agentops__a25f005e
• Fix: Ensure Node.js 22 install works on bugswarm base image

13. latex-formula-extraction

• Bug: Daytona SDK 0.168.0/0.169.0 SessionCommandLogsResponse pydantic validation error. Returns empty string instead of dict.
• Trial dir: latex-formula-extraction__266e48be
• Fix: Upstream Daytona SDK bug — report to Daytona

14. react-performance-debugging

• Bug: Verifier pytest-playwright page fixture not found. 5 tests SKIPPED (async), 6 tests ERROR. Agent's React optimizations never evaluated.
• Agent work: 21 tool calls, 7 code edits (parallelized API calls, added memoization, replaced lodash).
• Trial dir: react-performance-debugging__f89a18bd
• Fix: Fix conftest.py / pytest-playwright version compatibility in verifier

---

Additional Issues (not bugs, but score-impacting)

Disk space (2 tasks)

• seismic-phase-picking and video-filler-word-remover: Sandbox 74% full at start. Agent wastes 5+ minutes clearing pip cache before doing real work. Combined with tight timeouts, causes failures.
• Fix: Ensure sandbox images have sufficient free disk space

DinD Compose (4 tasks)

• gh-repo-analytics, pedestrian-traffic-counting, pg-essay-to-audiobook, scheduling-email-assistant all fail with Process closed stdout (rc=None) because SSH pipes break through DinD layers.
• Fix: PR on benchflow fix/dind-pty-acp branch — uses Daytona PTY WebSocket instead of SSH. Tested successfully on pedestrian-traffic-counting (reward=0.014, 7 tool calls).

---

Impact

Category	Tasks	Potential score gain
Verifier broken (false FAILs)	5	+5 passes
Sandbox permissions	4	+3-4 passes
Environment provisioning	2	+1-2 passes
Agent launch / SDK	3	+1-2 passes
Total	14	+10-13 passes

Current: 33.4/83 = 40.3%
Estimated if fixed: 43-46/83 = 52-55%

This would put Opus 4.7 above Gemini Flash (48.7%) on the leaderboard.

[xdotli]

Status update — audit against current main (post v0.4→v0.5 migration)

PR #309 (merged) fixes bug 2: [verifier] pytest_plugins in task.toml was being silently dropped because VerifierConfig had no such field, leaving lockdown._discover_pytest_plugin_flags's documented task.toml fallback as dead code. pytest_plugins: list[str] is now a real config field, plus a regression test on the real TaskConfig parse path. This also unblocks the task.toml-side remediation for bug 14 (declare pytest_plugins=["playwright"]).

The other 12 bugs do not have a root cause in src/benchflow/ and so cannot be fixed in this repo:

Task-repo bugs (need PRs against benchflow-ai/skillsbench — per-task test.sh / Dockerfile / datagen):

• Bug 1 — suricata-custom-exfil: test.sh installs pytest to wrong Python
• Bug 3 — sales-pivot-analysis: test.sh missing openpyxl
• Bug 4 — pddl-tpp-planning: reference task01.pkl not provisioned into tests/
• Bug 5 — fix-build-google-auto: uv not installed by the task Dockerfile
• Bug 6 — syzkaller-ppdev-syzlang: /opt/syzkaller/sys/linux/ root-owned (task Dockerfile)
• Bug 7 — taxonomy-tree-merge: /root/output/ root-owned + 369s build (task Dockerfile)
• Bug 8 — lean4-proof: /root/.elan/ toolchain not agent-accessible (task Dockerfile)
• Bug 9 — multilingual-video-dubbing: /outputs root-owned + verifier missing torch
• Bug 10 — organize-messy-files: input PDFs missing from Docker build context
• Bug 11 — azure-bgp-oscillation-route-leak: inline DATAGEN heredoc produces empty /app/data/
• Bug 12 — fix-build-agentops: claude-agent-acp rc=127 on bugswarm/cached-images base image PATH
• Bug 14 — react-performance-debugging: now resolvable by declaring pytest_plugins=["playwright"] in that task's task.toml (mechanism fixed by fix(verifier): restore [verifier] pytest_plugins task.toml field (#192 bug 2) #309)

Upstream bug:

• Bug 13 — latex-formula-extraction: Daytona SDK SessionCommandLogsResponse pydantic validation error — must be reported to / fixed by Daytona.

Keeping this issue open until the task-repo and upstream items are addressed.

[xdotli]

Task-repo bugs — skillsbench PR opened

Audited all 12 task-repo bugs (1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14) against current skillsbench main. 10 of them were already fixed by prior PRs (#733, #736, #764, #807, #835, #838). Only bugs 4 and 11 were still present.

benchflow-ai/skillsbench#846 fixes the two remaining bugs:

• Bug 4 — pddl-tpp-planning: verifier validate_plan() opened a <task>.pkl that instruction.md never asks the agent to produce → FileNotFoundError. Rewrote validate_plan() to parse the agent's text plan directly and validate it with unified_planning's SequentialPlanValidator. No .pkl required; any plan that solves the problem is accepted.
• Bug 11 — azure-bgp-oscillation-route-leak: inline RUN python3 <<'DATAGEN' heredoc left /app/data/ empty under Daytona's image build. Replaced with COPY + RUN of the already-present, identical generate_data.py (same pattern as the organize-messy-files fix in #764).

Daytona verification:

Task	Agent	Reward
pddl-tpp-planning	oracle	1.0
pddl-tpp-planning	claude-agent-acp (haiku-4-5)	1.0 — the exact audit failure case (agent writes text plan, no pkl) now passes
azure-bgp-oscillation-route-leak	oracle	1.0 — /app/data/ populated

skillsbench CI required gate (validate-tasks) is green on the PR.

Bug 13 (latex-formula-extraction, Daytona SDK) remains upstream-only.