Minor problems when trying benchmark

[tianyi-ge]

Hi, thanks for creating this interesting project! I'm trying to run metaclaw benchmark but meet some problems as follows

Wrong bench directory and extra argument

benchmark/README.md suggests

metaclaw-bench check data/metaclaw-bench/all_tests.json

but actually I need to run

metaclaw-bench check -p benchmark/data/metaclaw-bench/all_tests.json

The base directory of input path is not the current MetaClaw/benchmark, but MetaClaw

metaclaw-bench run usage is also updated, but doc is old

➜ metaclaw-bench run data/metaclaw-bench-small/all_tests.json --output results/
usage: metaclaw-bench run [-h] -i INPUT -o OUTPUT [-w WORKERS] [-n RETRY] [--scene-per-train SCENE_PER_TRAIN] [--memory] [--memory-proxy-port MEMORY_PROXY_PORT]
metaclaw-bench run: error: the following arguments are required: -i/--input

Hardcoded directory

Under MetaClaw/benchmark/scripts, many scripts are hardcoded to run under /home/xkaiwen

➜ grep -lrn "xkaiwen"
./openclaw_customize/llm-prompt-logger/index.ts
./scripts/dummy_run.py
./scripts/rl_only_run.py
./scripts/rl_only_memory_run.py
./scripts/memory_run.py
./scripts/baseline_run.py
./scripts/skills_only_run.py
./scripts/proxy_run.py
./scripts/rl_run.py
./scripts/skills_memory_run.py
./scripts/madmax_memory_run.py

Users have to replace all of them. Better use a BASE_DIR environment, or by default use the current directory + subdirectory path

Dummy run

To run dummy_run.py, env BENCHMARK_API_KEY is required, but as-is

# Use pre-built script
python scripts/dummy_run.py

# Manually run full pipeline (infer → score → report)
metaclaw start  # start metaclaw proxy first
export BENCHMARK_BASE_URL=http://127.0.0.1:30000/v1
export BENCHMARK_MODEL=GPT-5.2
metaclaw-bench run data/metaclaw-bench/all_tests.json --output results/

Confusing env var

I'm confused about what BENCHMARK_MODEL should be, and how it's used

Can this benchmark serve as an RL training set?

The benchmark looks like a comprehensive coding task. I wonder how it's generated, and if it's possible to use it as an RL training set?

Slow interrupt stop

It stucks for a while after ctrl-c

2026-04-07 20:47:07,031 | INFO     | metaclaw.launcher | [Launcher] signal 2 received — stopping …

I'd like to fix some easy problems if it's okay with you.

[kevinx0522]

Thanks for the detailed report — all five items are now fixed. Here is what changed.

---

Wrong bench directory and extra argument

benchmark/docs/CLI.md has been rewritten to match the actual CLI.

The correct invocations (run from the project root MetaClaw/):

metaclaw-bench check -p benchmark/data/metaclaw-bench/all_tests.json

metaclaw-bench run \
  -i benchmark/data/metaclaw-bench/all_tests.json \
  -o benchmark/results

check requires -p/--path; run requires -i/--input and -o/--output. The old positional-argument examples in the docs were stale.

---

Hardcoded directory

All hardcoded /home/xkaiwen paths have been removed from every script under benchmark/scripts/. Paths are now derived automatically from METACLAW_ROOT (env var) or from the script's own location (__file__):

_SCRIPT_DIR    = Path(__file__).resolve().parent
_METACLAW_ROOT = Path(os.environ.get("METACLAW_ROOT") or _SCRIPT_DIR.parent.parent)

The following env vars control path overrides:

Variable	Default
METACLAW_ROOT	auto-detected from scripts/ location
METACLAW_BENCH_BIN	metaclaw-bench on PATH
METACLAW_BIN	metaclaw on PATH
METACLAW_SKILLS_DIR	<METACLAW_ROOT>/memory_data/skills
METACLAW_API_KEY_SCRIPT	unset (use current env)

Full reference: benchmark/docs/env.md.

The hardcoded path in benchmark/openclaw_customize/llm-prompt-logger/index.ts (DEFAULT_LOG_DIR) is also fixed — it now reads METACLAW_ROOT first and falls back to $HOME/.metaclaw/llm_prompts.

---

dummy_run rename

benchmark/scripts/dummy_run.py is renamed to benchmark/scripts/proxy_passthrough_run.py.

The distinction between the two similar scripts is now explicit:

• baseline_run.py — no MetaClaw proxy at all; agent talks directly to BENCHMARK_BASE_URL.
• proxy_passthrough_run.py — proxy is started (using scripts/config/dummy.yaml) with all enhancements disabled; useful for validating proxy plumbing without any MetaClaw features active.

---

Missing BENCHMARK_API_KEY / confusing BENCHMARK_MODEL

benchmark/README.md now shows all three required env vars in Quick Start, with an explanation of what BENCHMARK_MODEL means.

A new benchmark/README.md#using-the-experiment-scripts section provides complete copy-paste examples for baseline, memory, skills, and RL modes.

benchmark/scripts/_env_arg_example.sh (moved from scripts/config/) is rewritten with per-variable comments and grouped sections (required / RL-specific / optional overrides).

---

Slow interrupt stop (Ctrl-C blocks)

benchmark/src/infer/infer_cmd.py now uses a timeout when waiting for the openclaw gateway to shut down after terminate():

gateway_proc.terminate()
try:
    await asyncio.wait_for(gateway_proc.wait(), timeout=8.0)
except asyncio.TimeoutError:
    gateway_proc.kill()
    await gateway_proc.wait()

Previously gateway_proc.wait() had no timeout and would block until the gateway finished its own cleanup.

---

New documentation

File	Contents
benchmark/docs/CLI.md	Full CLI reference, rewritten to match actual cli.py
benchmark/docs/scripts.md	Per-script guide, feature comparison table, env var overview
benchmark/docs/env.md	Complete env var reference — user-facing and internal (how each variable is consumed in the source)

---

Regarding the question "can the benchmark serve as an RL training set?" — yes, the benchmark tasks are multi-day interaction sequences with ground-truth preference signals. The all_tests_metaclaw.json variant is already used for online training in rl_*.py via --scene-per-train.

For offline data collection, baseline_run.py is the cleanest starting point: it runs the agent directly against your API with no proxy interference, so the collected trajectories reflect pure model behaviour.

Where to find trajectories:

• benchmark/results/<run>/*/infer_result.json — per-round result files written during inference. Each file covers the current (last) round only: the full user→assistant exchange including any tool calls and tool results for that round, plus the inline score. What is absent is the history from earlier rounds — prior user messages, assistant turns, and tool exchanges are not stored here.
• Full agent loop (all tool calls, internal reasoning, message history): two options:
  1. llm-prompt-logger plugin logs — benchmark/logs/llm_prompts/<session_id>/ (or the path set in openclaw.json → plugins.entries.llm-prompt-logger.config.logDir). Each *_round.json file contains the complete input/output for one LLM call including historyMessages.
  2. OpenClaw session JSONL files — benchmark/data/*/work/openclaw_state_<uuid>/agents/<agent_id>/sessions/<session_id>.jsonl. These are written by the openclaw gateway in real time and contain the raw turn-by-turn session transcript. The work/ directories are created fresh per run and can be cleaned up with metaclaw-bench clean -p benchmark/data.

---

Note on concurrency

Scripts that use the MetaClaw proxy (all scripts except baseline_run.py and proxy_passthrough_run.py) must run serially (-w 1, which is hardcoded in those scripts). Each test day carries state that the proxy accumulates — memory extracts, skill updates, or RL checkpoints — so day N+1 depends causally on day N.

baseline_run.py and proxy_passthrough_run.py can run in parallel (BENCH_WORKERS = 15 by default). With no MetaClaw state being accumulated, each day/scene is causally independent of the others; only the rounds within a single scene are sequential.

---

Feel free to open a PR for any of the above — the easy fixes (docs, env vars) would be straightforward to split out if preferred.

[tianyi-ge]

Thank you!!