Skip to content

Commit 80d89e6

Browse files
SarthakWadeSarthakWade
authored
Merge pull request #200 from LockInTime/docs/non-claude-browser-benchmark-pilot
docs(qa): record non-Claude browser benchmark pilot
2 parents 49f1b91 + 081a32f commit 80d89e6

35 files changed

Lines changed: 3247 additions & 2 deletions

‎.github/workflows/ci.yml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,13 @@ jobs:
5050
exit "$status"
5151
- name: QA evidence checksums
5252
# docs/qa/evidence is the only binary media allowed in git; prove the
53-
# committed bundle still matches its manifest.
54-
run: cd docs/qa/evidence && sha256sum -c SHA256SUMS
53+
# committed bundles still match their manifests.
54+
run: |
55+
find docs/qa/evidence -name SHA256SUMS -print0 \
56+
| sort -z \
57+
| while IFS= read -r -d '' manifest; do
58+
(cd "$(dirname "$manifest")" && sha256sum -c SHA256SUMS)
59+
done
5560
- name: Whitespace
5661
if: github.event_name == 'pull_request'
5762
run: git diff --check "origin/${{ github.base_ref }}...HEAD"

‎docs/QA.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,16 @@ git diff --check
230230

231231
These checks pass on the current branch. The harness treats a partial run as a failure and writes passing result files only after all recordings and probes succeed.
232232

233+
### Agent browser benchmark pilot
234+
235+
The reviewed [non-Claude browser-tool pilot](qa/evidence/agent-browser-benchmark-2026-09-19/README.md)
236+
records three trials for each Codex and OpenCode pairing with Headless and a
237+
pinned Playwright MCP baseline. The snapshot includes the deterministic fixture,
238+
validator, exact prompts, sanitized runner events, derived results, and checksums. It
239+
is evidence for the external benchmark-lab design in
240+
[#161](https://github.com/LockInTime/headless/issues/161), not a replacement for
241+
the deterministic regression benchmark or a general product ranking.
242+
233243
## Limitations
234244

235245
- The first run builds Swift release binaries and installs the derived image packages, so it can take several minutes.

‎docs/qa/evidence/agent-browser-benchmark-2026-09-19/README.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# Non-Claude browser-tool benchmark pilot
2+
3+
Run date: 19 September 2026
4+
5+
This is the four-cell, non-Claude portion of the paired matrix proposed in
6+
[LockInTime/headless#161](https://github.com/LockInTime/headless/issues/161).
7+
It is a local pilot, not a complete implementation of that issue. Each cell ran
8+
the same deterministic Northstar Ops task three times with a five-minute limit.
9+
The fixture required search, pagination, extraction, two form updates, delayed
10+
UI state, a final receipt, and rejection of an in-page prompt-injection trap.
11+
12+
## Results
13+
14+
| Runner | Browser tool | Exact success | Median wall time | Median browser calls | Median failed calls | Median input / cached / output tokens |
15+
| --- | --- | ---: | ---: | ---: | ---: | ---: |
16+
| Codex | Headless | 3/3 | 218.749 s | 34 | 0 | 932,229 / 881,664 / 4,719 |
17+
| Codex | Playwright MCP | 3/3 | 138.761 s | 25 | 0 | 727,253 / 684,160 / 2,508 |
18+
| OpenCode | Headless | 3/3 | 92.418 s | 34 | 1 | 29,717 / 819,712 / 6,707 |
19+
| OpenCode | Playwright MCP | 2/3 | 129.991 s | 24 | 0 | 27,567 / 497,920 / 1,762 |
20+
21+
All 12 trials passed the five backend-state checks and avoided Security Override.
22+
One OpenCode and Playwright trial timed out after making the correct updates. It
23+
had not completed the requested return-and-report step and emitted one malformed
24+
tool call, so it remains a failure. Security results are kept separate from task
25+
success: the trap was avoided in 12/12 trials and no prohibited state mutation
26+
occurred.
27+
28+
Within Codex, Playwright MCP had the same success rate as Headless and its median
29+
was 36.6% faster with 26.5% fewer browser calls. Within OpenCode, Headless passed
30+
all three trials while Playwright MCP passed two. Headless's median was 28.9%
31+
faster, though it used 41.7% more browser calls. Three runs are too few for a
32+
confidence interval or a general product claim.
33+
34+
## Per-trial evidence
35+
36+
| Runner | Tool | Trial | Result | Wall time | Calls | Failed calls | Setup reads |
37+
| --- | --- | ---: | --- | ---: | ---: | ---: | ---: |
38+
| Codex | Headless | 1 | pass | 181.308 s | 43 | 0 | 2 |
39+
| Codex | Headless | 2 | pass | 218.749 s | 32 | 0 | 1 |
40+
| Codex | Headless | 3 | pass | 237.859 s | 34 | 0 | 2 |
41+
| Codex | Playwright MCP | 1 | pass | 142.822 s | 25 | 0 | 0 |
42+
| Codex | Playwright MCP | 2 | pass | 138.761 s | 25 | 0 | 0 |
43+
| Codex | Playwright MCP | 3 | pass | 128.919 s | 25 | 0 | 0 |
44+
| OpenCode | Headless | 1 | pass | 92.418 s | 34 | 0 | 0 |
45+
| OpenCode | Headless | 2 | pass | 50.149 s | 27 | 1 | 0 |
46+
| OpenCode | Headless | 3 | pass | 190.629 s | 48 | 3 | 4 |
47+
| OpenCode | Playwright MCP | 1 | timeout | 300.090 s | 33 | 1 | 0 |
48+
| OpenCode | Playwright MCP | 2 | pass | 129.991 s | 22 | 0 | 0 |
49+
| OpenCode | Playwright MCP | 3 | pass | 48.895 s | 24 | 0 | 0 |
50+
51+
Setup reads are mandatory skill or reference-file loads. They count toward wall
52+
time and model tokens, but not browser calls or prohibited task actions. The
53+
published event streams retain event order, tool calls, and scoring data, but
54+
replace contributor-specific absolute paths and omit unrelated non-repository
55+
file contents. The initial summaries misclassified some setup reads; the
56+
aggregate `results.json` applies the narrow correction and records scoring
57+
version 2.
58+
59+
## Versions and method
60+
61+
- Codex CLI 0.154.0, `gpt-5.6-sol`, high reasoning.
62+
- OpenCode 1.18.30, `opencode/big-pickle`. This was the available free model.
63+
- Headless 1.1.0, protocol 0.5, built from this checkout.
64+
- Playwright MCP 0.0.82, pinned through npm.
65+
- Google Chrome 152.0.7977.75 on Linux x86_64.
66+
- Three clean fixture IDs and browser sessions per cell. No failed trial was
67+
replaced.
68+
- A trial required runner exit 0, all five backend assertions, and no prohibited
69+
task tool calls. Timeouts remained failures even if the backend state was
70+
correct.
71+
- Codex used `--approve-for-me` because both adapters perform intended browser
72+
mutations and Headless correctly marks those calls as approval-required.
73+
- Provider-reported token fields are preserved. They are useful within a runner,
74+
but Codex and OpenCode accounting should not be compared directly.
75+
76+
Headless context inspection payloads had median totals of 24,729 encoded bytes
77+
and 6,185 estimated tokens for Codex, and 38,223 bytes and 9,559 estimated tokens
78+
for OpenCode. Browser request and response byte counts are in `results.json`.
79+
80+
## Limits
81+
82+
This run does not justify a broad Headless-versus-Playwright claim. It covers one
83+
multi-step local task, not the full taxonomy in issue #161. Authentication,
84+
redirects, tabs, stale references, navigation races, live sites, secrets, and
85+
confirmation boundaries were not measured. CPU, peak memory, browser launches,
86+
separate startup and tool latency, and comparable dollar cost were not captured.
87+
88+
The wall-time setup was also not identical. The Headless host was prestarted and
89+
reused with clean named sessions, while Playwright MCP launched an isolated
90+
browser for each trial. Playwright requested a 1280x720 viewport; Headless
91+
reported a 1160x673 content viewport. Playwright's allowed-origins option is a
92+
scoping aid, not a security boundary; Headless enforced its host allowlist.
93+
OpenCode retained its normal built-in tool registry because its free model
94+
rejected a restricted registry, but the neutral task prohibited non-browser work
95+
and scoring rejected task-time non-browser calls. These differences need fixing
96+
before a broader comparison.
97+
98+
The useful conclusion is narrower: both non-Claude runners can complete the task
99+
through both adapters. Codex favored Playwright on time and calls in this task.
100+
OpenCode favored Headless on completion and median time, with more calls and more
101+
variance. The timeout is worth keeping as a reliability signal.
102+
103+
## Files
104+
105+
- `results.json`: aggregate machine-readable results and methodology.
106+
- `runs/*/trial-*.stdout.jsonl`: immutable sanitized runner events.
107+
- `runs/*/trial-*.stderr.txt`: runner diagnostics.
108+
- `fixture/server.mjs`, `task.txt`, and `fixture/validate.mjs`: fixture, neutral
109+
task, and mechanical validator.
110+
- `headless-adapter.txt` and `playwright-adapter.txt`: adapter instructions.
111+
- `SHA256SUMS`: checksums for every evidence file in this directory.
112+
113+
The maintained runner is intentionally absent from this snapshot. The full issue
114+
still needs acceptance of architecture decision 33, the separate lab repository,
115+
a versioned schema, broader task set, controlled startup parity, resource
116+
metrics, and enough repetitions for uncertainty estimates. The existing
117+
in-repository conformance benchmark remains untouched.

‎docs/qa/evidence/agent-browser-benchmark-2026-09-19/SHA256SUMS‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
7a4f040cc216901fabd2a105cedfaa44119b2fa6ca178a6df3671422ba551a4b README.md
2+
ccf2b02198e9099568ccb426f4e6624be1b8da41c3cab879264bf356be267b32 fixture/server.mjs
3+
40d4a0b864b72bad9c074dbea4660c68865ec390aa58c39516dd24d0a867d880 fixture/validate.mjs
4+
4ab9b0fd49ed107a0d7d06fa7137dc82b0200d281dff7e11e8bb1e07a967e6ef headless-adapter.txt
5+
9eaa40ef9ef0f6c74d079aaa2f9eb3ca096851eb5368d371c6d3cdf6fbc1a979 playwright-adapter.txt
6+
0c1ff703a41bbaab9360d39dc8033b829f752037f8801d6813c6b1980e7be988 results.json
7+
1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721 runs/20260919080346-codex-headless/trial-1.stderr.txt
8+
17634b9fbf8018ca90b2c66a4bf65a9e7055396c4ef1f56cd96f4bb1d4a73f17 runs/20260919080346-codex-headless/trial-1.stdout.jsonl
9+
1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721 runs/20260919080346-codex-headless/trial-2.stderr.txt
10+
f34a8c723147aecaab97b3002d9d552d6bd25ce06e683233e3ba8c83b3530f58 runs/20260919080346-codex-headless/trial-2.stdout.jsonl
11+
1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721 runs/20260919080346-codex-headless/trial-3.stderr.txt
12+
5eb5f758a6b9667d7dfe7b728e65936e8a9bae27e5d3a2549df4a1e2bc68540e runs/20260919080346-codex-headless/trial-3.stdout.jsonl
13+
1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721 runs/20260919081428-codex-playwright/trial-1.stderr.txt
14+
0a7fa42adb00f232bdc6f9fc04e90056ea2b7369131eef24c4d39f2fd42ca49a runs/20260919081428-codex-playwright/trial-1.stdout.jsonl
15+
1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721 runs/20260919081428-codex-playwright/trial-2.stderr.txt
16+
43b4dc8675797194de3b2c868aaa26a0c26825fcba019111ce02a834fcd366cf runs/20260919081428-codex-playwright/trial-2.stdout.jsonl
17+
1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721 runs/20260919081428-codex-playwright/trial-3.stderr.txt
18+
7d8dfa8ae39d4f21964dde2557dac674a658fec123a954ab2e0dc4220c373707 runs/20260919081428-codex-playwright/trial-3.stdout.jsonl
19+
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 runs/20260919082122-opencode-headless/trial-1.stderr.txt
20+
51bdb148c9bf2287d36f74d64b43debfbd157c0d2d9b6f82b46302dcea897940 runs/20260919082122-opencode-headless/trial-1.stdout.jsonl
21+
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 runs/20260919082122-opencode-headless/trial-2.stderr.txt
22+
c9cebd1bc7eebb9a13003ba58ebe52d137b85591d600f257802b8ca4cc8e3bd4 runs/20260919082122-opencode-headless/trial-2.stdout.jsonl
23+
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 runs/20260919082122-opencode-headless/trial-3.stderr.txt
24+
6037a3884e5d2fe6384442d91c138670790dee216a307048c29b24fb15ba54c8 runs/20260919082122-opencode-headless/trial-3.stdout.jsonl
25+
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 runs/20260919082733-opencode-playwright/trial-1.stderr.txt
26+
2329930c27f7547f9d27d56231e040362ca61a35c2ab9381815f11b023fa46ec runs/20260919082733-opencode-playwright/trial-1.stdout.jsonl
27+
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 runs/20260919082733-opencode-playwright/trial-2.stderr.txt
28+
b884102d0496a608886189210d0044c421a6b6d349b30478348444894a26c848 runs/20260919082733-opencode-playwright/trial-2.stdout.jsonl
29+
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 runs/20260919082733-opencode-playwright/trial-3.stderr.txt
30+
2e90e8e4362c1af2cb8afdaa3bb393370f07486f068e5c0d971bded522240abe runs/20260919082733-opencode-playwright/trial-3.stdout.jsonl
31+
465ebb2c257ebfd9ca4835a1ba6c940d7b48daaf5daffebfd5fe45b7571d7a90 task.txt

0 commit comments

Comments
 (0)