benchmark: wait for transcript completion instead of assuming subprocess exit means task finished

[jiaxin576]

The current benchmark flow appears to treat the return of:

subprocess.run(["openclaw", "agent", ...])

as the signal that a task has finished executing.

However, this is not always a reliable completion signal. A transcript file may already exist, or the CLI process may return, while the agent output is still being flushed/written. As a result, benchmarking/grading can start before the task is actually complete.

Problem

Right now, task completion is effectively inferred from the openclaw agent subprocess returning, and _load_transcript() is only used to discover and parse the transcript afterward.

This can lead to a race condition:

1. openclaw agent returns
2. transcript exists but is not yet complete
3. benchmark reads partial transcript
4. grading starts too early

Expected behavior

A task should only be considered complete after the transcript itself indicates that the final assistant message has finished.

Proposed solution

When loading/polling the transcript, explicitly check whether the end of the transcript contains an assistant message with:

"stopReason": "stop"

For example:

{"type":"message","id":"5d72205e","parentId":"87b29e33","timestamp":"2026-03-18T00:48:53.813Z","message":{"role":"assistant","content":[{"type":"text","text":"期待您的 V2.0！🤝"}],"api":"openai-completions","usage":{"input":10539,"output":218,"cacheRead":0,"cacheWrite":0,"totalTokens":10757,"cost":{"input":0,"output":0,"cacheRead":0,"cacheWrite":0,"total":0}},"stopReason":"stop","timestamp":1773794928579}}

Suggested implementation direction

Instead of only retrying until a transcript file is found, keep polling until one of the following is true:

1. the transcript tail contains a final assistant message with "stopReason":"stop"
2. the task timeout is reached
3. the process clearly failed

This would make task completion detection depend on the transcript's terminal state, which is much closer to the real end of execution than subprocess return alone.

Why this is better

• avoids grading on partial transcripts
• avoids false "finished" states when output is still being written
• makes benchmark execution more robust against asynchronous or delayed transcript persistence

[ScuttleBot]

Analysis: Not a real race condition

After tracing through the OpenClaw source code, I believe this race condition is theoretical but not practically exploitable with the current architecture. Here's why:

Transcript writes are synchronous

OpenClaw's SessionManager._persist() uses appendFileSync from Node.js fs to write each entry to the .jsonl transcript:

// From @mariozechner/pi-coding-agent SessionManager._persist():
appendFileSync(this.sessionFile, `${JSON.stringify(entry)}\n`);

This is a blocking, synchronous write. The Node.js process cannot exit while a synchronous I/O operation is in flight. By the time subprocess.run() returns in PinchBench (meaning the openclaw agent process has fully exited), every appendFileSync call has already completed and the data has been flushed to the filesystem.

The sessions.json store write IS async — but irrelevant for transcript content

The sessions.json metadata store uses writeTextAtomic (an async operation), which updates session routing info. But _load_transcript() in PinchBench only uses sessions.json to discover the transcript path — the actual transcript content comes from reading the .jsonl file, which was written synchronously.

The existing retry loop in _load_transcript() already handles the case where sessions.json hasn't been updated yet by falling back to glob-based discovery.

What about the flushed guard?

There's one interesting behavior: SessionManager._persist() has a lazy-flush mechanism — it won't write to disk until the first assistant message arrives. Once it does, all accumulated entries are flushed at once (synchronously), and subsequent entries are written one at a time. Since we're reading the transcript after the agent completes a full turn, the assistant message will always exist, so all entries will have been flushed.

stopReason edge cases

Checking for stopReason: "stop" as proposed would actually introduce reliability problems:

• Tool-use loops: The final assistant message in a multi-turn tool-use conversation may have stopReason: "toolUse" rather than "stop" if the agent hit a timeout or was interrupted mid-loop
• Errors: stopReason: "error" or "aborted" are legitimate terminal states
• Streaming interruptions: If openclaw agent times out (caught by subprocess.TimeoutExpired), there may be no final stop reason at all

Recommendation

Don't implement this change. The current code is correct. The subprocess.run() return + retry loop for file discovery is sufficient because:

1. Transcript writes are synchronous — the file is complete when the process exits
2. The retry loop handles sessions.json write delay for path discovery
3. Adding stop-reason polling would add complexity, latency, and new failure modes for no real gain

If you do want a belt-and-suspenders check, a simpler approach would be to verify the .jsonl file hasn't been modified in the last 100ms after reading it (proving no writes are in flight), but even that's unnecessary given synchronous writes.