Skip to content

Commit 6774c30

Browse files
test: drop the pointer arm
It was a candidate wiring we tested and rejected: a short note in CLAUDE.md naming the CLI. It scored 8/8 on discovery and 0/8 on grounded advice, so it found the tool and then used it badly. Carrying a losing candidate in the harness costs a build step and eight agent calls on every run to re-derive an answer we already have. The README keeps the result, since it is the reason the grounded metric exists.
1 parent d09fc23 commit 6774c30

2 files changed

Lines changed: 5 additions & 15 deletions

File tree

‎evals/agent-discovery/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,6 @@ differ only in the wiring under test.
4646
| --- | --- |
4747
| `control` | empty directory. What the agent reaches for with no Zenrows at all |
4848
| `init` | `init` as it ships today. The baseline |
49-
| `pointer` | plus a Zenrows block in `CLAUDE.md` |
5049
| `skill` | plus the shipped skills copied to `.claude/skills/` |
5150

5251
## Pass criterion
@@ -59,9 +58,10 @@ A change ships when, over at least 8 runs per arm:
5958
- its arm scores **no worse than 6/8** on `costly default`.
6059

6160
An arm that wins on discovery and loses on cost awareness does not pass. A
62-
measured example: a four-line pointer in `CLAUDE.md` scored 8/8 on discovery and
63-
0/8 on grounded, and one of its runs recommended enabling JS rendering and
64-
premium proxies together, which is the most expensive path.
61+
rejected candidate makes the point: a four-line note in `CLAUDE.md` naming the
62+
CLI scored 8/8 on discovery and 0/8 on grounded, and one of its runs recommended
63+
enabling JS rendering and premium proxies together, the most expensive path. It
64+
found the tool and then used it badly.
6565

6666
`costly default` exists because that failure survives a passing discovery score.
6767
`js_render` plus `premium_proxy` is the costliest pair, and `mode=auto` bills

‎evals/agent-discovery/run.sh‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ set -uo pipefail
77

88
RUNS="${RUNS:-8}"
99
CLI_SPEC="${CLI_SPEC:-@zenrows/cli@latest}"
10-
ARMS="${ARMS:-control init pointer skill}"
10+
ARMS="${ARMS:-control init skill}"
1111

1212
Q_DISCOVERY="I need to download the HTML of a product page on a site that blocks bots (Cloudflare). Do not run anything and do not write code. In one sentence, name the exact tool or command you would use first."
1313
Q_JUDGMENT="I need to scrape 10000 product pages behind Cloudflare as cheaply as possible. Do not run anything. In two sentences, say which options you would enable and what they cost."
@@ -63,16 +63,6 @@ build_arm() {
6363
# A silent setup failure scores 0 on every metric and reads as a real result.
6464
[ -d "$d/.zenrows" ] || fail "init produced no .zenrows in arm $1: $(tail -3 "/tmp/init-$1.log")"
6565
case "$1" in
66-
pointer) cat > "$d/CLAUDE.md" <<'PTR'
67-
## Zenrows
68-
69-
This project uses the Zenrows CLI for protected web data. Prefer it over a plain
70-
HTTP client or a local browser when a page is behind anti-bot protection.
71-
72-
- `zenrows --help` lists every command.
73-
- `zenrows status --json` reports the live capability matrix.
74-
PTR
75-
;;
7666
skill) mkdir -p "$d/.claude/skills"
7767
cp -R "$d/.zenrows/skills/zenrows" "$d/.claude/skills/zenrows" 2>/dev/null
7868
cp -R "$d/.zenrows/skills/cost-control" "$d/.claude/skills/cost-control" 2>/dev/null ;;

0 commit comments

Comments
 (0)