Skip to content

README.md

Latest commit

 

History

History
229 lines (178 loc) · 9.9 KB

README.md

File metadata and controls

229 lines (178 loc) · 9.9 KB
Raw

cbrun — CodingBench Solver Benchmark Runner

cbrun is a Harbor-independent benchmark runner. It evaluates an autonomous coding agent by giving it only the public task specification and letting it build the project from scratch, then scoring the result against hidden acceptance tests.

Task model (solver-only)

The agent's job is to write the implementation. The test code is a hidden benchmark asset, never an agent output.

  • Input the agent sees: the PRD (/environment/prd/Full_PRD.md) and the Interface Contract (/environment/Interface_Contract.md), both also embedded in the instruction prompt. The agent's workspace /app starts empty.
  • Output the agent produces: the implementation in /app, importable / installable from the workspace root.
  • Scoring: after the agent finishes, the runner injects the hidden acceptance test suite and runs it; the reward is binary (1 = all pass, 0 = otherwise).

The guiding invariant is information completeness: the provided environment (empty /app + PRD + Interface Contract) must be theoretically sufficient for an ideal agent to pass the hidden tests. If a case's test_command depends on build scaffolding the spec never declares, that is a case spec defect to fix in the case assets — never patched in the runner.

Fairness: visible checks vs hidden gates

  • Visible / dev checks: the agent may write and run its own tests and any public sample checks as many times as it wants and read their full output. A real develop → test → debug loop is expected.
  • Hidden gates: the hidden acceptance tests are physically absent from the solve container. cbrun derives a per-case :agent image from the published :deliverable image and removes /tests/final in a new image layer (the tests are extracted to a host-side cache first). They are re-injected only for the judge phase, and judge output is not fed back to the agent.

Free development, time-limited

There is no limit on steps, turns, edits or cost. The only hard ceilings are wall-clock budgets:

  • max_agent_timeout_sec — solve phase. Final-stage cases use a uniform 2h (7200s).
  • max_test_timeout_sec — judge phase. Default 10min (600s), per-case overridable.
  • --timeout-multiplier scales both wall clocks (mirrors Terminal-Bench's global_timeout_multiplier).

A conservative stall watchdog stops a wedged CLI without misjudging a long compile or long model turn as a hang.

Termination (no false positives)

The judge always runs against the final workspace state regardless of how the solve ended, so the reward never depends on an agent self-report. The solve terminal status is recorded as one of:

  • completed — the CLI process exited 0 on its own (the implicit "submit").
  • timeout — the wall clock was hit.
  • error — stall-killed or a non-zero CLI exit (including agent setup failure).

Built-in backends

Three backends ship as built-in AgentSpec records:

Backend CLI Notes
codex @openai/codex Writes ~/.codex/auth.json + config.toml from OPENAI_* env before solve. Default model_prefix=keep for OpenAI-compatible gateways.
opencode opencode-ai Forwards provider env based on provider/model id.
claude-code @anthropic-ai/claude-code Runs as non-root user cbagent with bypassPermissions (Claude Code rejects root). Judge still runs as root.

Auth/provider env is selected centrally and forwarded into the container; only present keys are injected (key names are recorded in results, never values). CLI versions default to @latest but should be pinned for reproducibility via CBRUN_CODEX_VERSION / CBRUN_OPENCODE_VERSION / CBRUN_CLAUDE_VERSION. On a shared Docker daemon, set CBRUN_IMAGE_NAMESPACE to keep this release's caseNNN:{deliverable,agent} tags separate from older case-number mappings. The default remains codingbench-benchmark.

Codex + OpenAI-compatible gateway

Point standard env vars at any OpenAI-compatible endpoint:

export OPENAI_API_KEY=your-key
export OPENAI_BASE_URL=https://your-gateway.example/v1
cbrun --case case010 --backend codex --model openai/gpt-4o-mini

cbrun writes a dedicated model provider into ~/.codex/config.toml during setup. It uses the Responses API over HTTPS and disables WebSocket transport, because OpenAI-compatible gateways do not necessarily implement WebSockets. Use model_prefix: strip in a custom spec if your Codex install expects leaf model names only.

Claude Code non-root contract

  • Agent deliverables must land in /app (hard contract; not overridable).
  • Solve runs as cbagent; judge reads /app as root.
  • Agent HOME (/home/cbagent) holds CLI caches only; it is not scored.

Bring your own agent

Pass a local AgentSpec file instead of --backend:

cbrun --case case010 --agent-spec ./my-agent.json --model my/model

Example my-agent.json:

{
  "name": "my-agent",
  "env_passthrough": ["MY_API_KEY"],
  "setup_script": "mkdir -p \"$HOME/.myagent\" && echo ok > \"$HOME/.myagent/ready\"",
  "command": "my-cli --model {model_quoted} --workdir {workdir_quoted} \"$(cat {instruction_quoted})\" 2>&1 | tee {log_quoted}",
  "run_as": "root",
  "model_prefix": "keep",
  "setup_timeout_sec": 120
}

AgentSpec fields

Field Required Description
name yes Identifier recorded in results.
command yes Shell command template for the solve phase. Workdir is always /app.
env_passthrough no Env var names to forward when present on the host.
setup_script no Shell run before solve (auth files, config). Do not use set -x.
install_script no Optional CLI install when not baked into the :agent image.
run_as no root (default) or nonroot (cbagent via docker exec -u).
model_prefix no keep (default) or strip the provider/ prefix from --model.
home no Writable HOME/config directory (defaults by run_as).
setup_timeout_sec no Setup phase timeout (default 120s).
python_hook no Built-in hooks only (cbrun.agent_hooks:*); custom specs should use declarative fields.

Placeholders in command / setup_script: {model}, {model_quoted}, {instruction_quoted}, {log_quoted}, {workdir} (/app), {workdir_quoted}, {home}.

Security boundaries

  • AgentSpec files are local trusted configuration. They can execute shell (setup_script, install_script, command). Do not run specs from untrusted sources.
  • Specs are loaded from --agent-spec PATH only; cbrun does not fetch and execute remote spec URLs.
  • The solve container never mounts the Docker socket.
  • Hidden tests are absent during solve; logs must not contain secret values.

Network policy and upstream denylist

  • During solve, GitHub hostnames are blocked via container /etc/hosts (--add-host …:0.0.0.0). Model API calls and PyPI/npm remain reachable. Disable with --no-block-github.
  • Each case may ship source/denylist.json listing upstream package/module names for the target product. pip/conda install shims in the :agent image reject those packages inline (warning only, no scoring).
  • After solve, cbrun statically scans /app for banned imports/symbols. If found, the agent gets one fix retry (--denylist-fix-retries, default 1); if violations remain, the trial scores reward=0. Disable with --no-enforce-denylist.

Step mode (architecture only)

Usage

# One case, one backend
cbrun --case case010 --backend codex --model openai/gpt-5.5

# Several backends, reward matrix
cbrun --case case010 \
  --backend codex --backend opencode --backend claude-code \
  --model openai/gpt-5.5

# Custom agent spec
cbrun --case case010 --agent-spec ./agents/echo.json --model dummy/model

# Every case under the cases root
cbrun --all --backend codex --model openai/gpt-5.5

Outputs go to --out (default benchmark/output/cbrun/): per-trial agent.log, agent_setup.log (when setup runs), judge.log, final_report.json, plus an aggregate summary.json and a printed reward matrix.

Each trial records reproducibility metadata: agent spec name/hash, resolved model, run_as, model_prefix, setup status, forwarded env key list, and CLI version when available.

Trial phases

  1. Derive/reuse the :agent image (extract hidden tests to host cache, install pinned CLIs, create cbagent user, rm -rf /tests/final). Idempotent.
  2. Start a solve container (--gpus only when the case needs it, host network for model APIs, never the Docker socket).
  3. Inject the instruction; setup agent auth/config; chown /app (+ HOME for non-root agents); run the agent in /app under the wall clock + stall watchdog, tee'ing output to agent.log.
  4. Decide the terminal status; ensure no agent process lingers.
  5. Re-inject hidden tests + a synthesized /task.toml, run the shared final_judge.py as root, parse the binary reward (distinguishing judge_error).

Reused components

  • coding_bench_harbor.adapter: case discovery, CaseAssets, the from-scratch _INSTRUCTION_PREAMBLE, runner normalization.
  • coding_bench_harbor.final_judge: the scoring engine (single source of truth, shared with the Harbor adapter).

Tests

  • Fast unit tests (no Docker): tests/test_cbrun_agents.py, tests/test_cbrun_agent_spec.py, tests/test_cbrun_core.py, tests/test_cbrun_wiring.py.
  • Docker-gated integration (@pytest.mark.slow, tests/test_cbrun_docker.py): oracle sanity (GT → reward 1), fairness invariant (:agent image has no /tests/final, CLIs present, cbagent user exists, Codex setup writes config). They skip automatically when Docker or the required images are unavailable.
  • Local agent smoke (your credentials, no source edits): ../local_agents/README.md — copy *.env.example, run ./local_agents/run_smoke.sh.