🧪 TestSuiteBench

🏆 Leaderboard | 📝 Paper | 🤗 Dataset

🤔 Can coding agents write a test suite strong enough to tell a correct program from a subtly broken one, working only from its specification? Each task pairs a specification of a real CLI tool (from ProgramBench) with the ground-truth executable and a set of mutants.

🎯 A generated test suite is scored on two aspects:

• mutation score: the fraction of mutants killed by the test suite.
• validity rate the fraction of test functions that passed reference implementation.

⚙️ Setup

Important

🐳 All task images are built for linux/amd64. Run construction and evaluation on a Linux x86_64 machine with Docker.

uv sync

The final specifications are already in ground_truth/spec/<instance_id>/spec.md. The reference tests, mutants, and ground-truth executables are read directly from HuggingFace (Reference Tests, Mutants, Ground Truth): each task downloads only its own files into the HuggingFace cache when it first needs them.

Optionally, pre-download the datasets into the cache, for example before running offline with HF_HUB_OFFLINE=1:

# [!WARNING] This is optional and takes ~100 GB of your storage
uv run hf download eth-sri/tsb-reference_tests --repo-type dataset
uv run hf download eth-sri/tsb-mutants --repo-type dataset
uv run hf download eth-sri/tsb-reference_impl --repo-type dataset

🔑 Agent credentials

Log in or export the credentials for the agent you want to run before launching it.

codex login                         # Codex
export OPENAI_API_KEY=...           # Codex (API)
export CLAUDE_CODE_OAUTH_TOKEN=...  # Claude Code
export GEMINI_API_KEY=...           # Gemini CLI
export OPENROUTER_API_KEY=...       # OpenRouter-based models

Agent	Credential
Codex	Read from ~/.codex/auth.json or OPENAI_API_KEY.
Claude Code	1. CLAUDE_CODE_OAUTH_TOKEN, a long-lived token from claude setup-token. 2. ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN. 3. The claude login in ~/.claude/.credentials.json.
Gemini CLI	GEMINI_API_KEY or the gemini login in ~/.gemini/oauth_creds.json.
OpenRouter-based models	OPENROUTER_API_KEY, exported to the agent as an OpenAI-compatible key.

📊 Evaluation

scripts/eval/run_agents.sh runs one or more model configurations (--runs) over a set of tasks. Agents run in Docker with network access limited to their provider's API. The available run IDs and their harnesses are listed in scripts/agents/README.md.

🧬 Test Generation

Generate a test suite from the specification and evaluate it against the ground truth and the mutants:

bash scripts/eval/run_agents.sh --runs "claude-opus-5 codex-gpt-5.6-sol" \
    --tasks-file ground_truth/valid_task.txt --jobs 2 -- --part test_gen

# Results are written to `output/eval/testgen/<model>_<reasoning>/<instance_id>/`

For all options of run_agents.sh (runs, tasks, parallelism, input locations, model and reasoning overrides), see docs/config.md.

uv run programbench eval-tests \
    output/eval/testgen/<model>_<reasoning> \
    --workers 4 --mutant-workers 8

Each <instance_id>/tests.tar.gz must unpack to an eval/run.sh that runs the tests against ./executable and writes JUnit XML to eval/results.xml.

💻 Code Generation

Have the agent implement the program from the same specification and evaluate it against the reference tests:

bash scripts/eval/run_agents.sh --runs "claude-opus-5 codex-gpt-5.6-sol" \
    --tasks-file ground_truth/valid_task.txt --jobs 2 -- --part exec_gen

# Results are written to `output/eval/codegen/<model>_<reasoning>/<instance_id>/`

🗂️ Repository Layout

The inner workings of the pipeline are detailed in docs/explore.md.

📚 Citation

🙏 If you find our work helpful, please cite:

@misc{egashira2026testsuitebench,
      title={TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications},
      author={Kazuki Egashira and Niels Mündler-Sasahara and Mark Niklas Müller and Martin Vechev},
      year={2026},
      eprint={2610.13065},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2610.13065},
}

💡 Please also consider citing ProgramBench, which inspired our work and forms the basis of this code base:

@misc{yang2026programbench,
    title={ProgramBench: Can Language Models Rebuild Programs From Scratch?},
    author={John Yang and Kilian Lieret and Jeffrey Ma and Parth Thakkar and Dmitrii Pedchenko and Sten Sootla and Emily McMilin and Pengcheng Yin and Rui Hou and Gabriel Synnaeve and Diyi Yang and Ofir Press},
    year={2026},
    eprint={2605.03546},
    archivePrefix={arXiv},
    primaryClass={cs.SE},
    url={https://arxiv.org/abs/2605.03546},
}

📄 License

MIT LICENSE