You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It was a candidate wiring we tested and rejected: a short note in CLAUDE.md
naming the CLI. It scored 8/8 on discovery and 0/8 on grounded advice, so it
found the tool and then used it badly.
Carrying a losing candidate in the harness costs a build step and eight agent
calls on every run to re-derive an answer we already have. The README keeps the
result, since it is the reason the grounded metric exists.
Copy file name to clipboardExpand all lines: evals/agent-discovery/run.sh
+1-11Lines changed: 1 addition & 11 deletions
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ set -uo pipefail
7
7
8
8
RUNS="${RUNS:-8}"
9
9
CLI_SPEC="${CLI_SPEC:-@zenrows/cli@latest}"
10
-
ARMS="${ARMS:-control init pointer skill}"
10
+
ARMS="${ARMS:-control init skill}"
11
11
12
12
Q_DISCOVERY="I need to download the HTML of a product page on a site that blocks bots (Cloudflare). Do not run anything and do not write code. In one sentence, name the exact tool or command you would use first."
13
13
Q_JUDGMENT="I need to scrape 10000 product pages behind Cloudflare as cheaply as possible. Do not run anything. In two sentences, say which options you would enable and what they cost."
@@ -63,16 +63,6 @@ build_arm() {
63
63
# A silent setup failure scores 0 on every metric and reads as a real result.
64
64
[ -d"$d/.zenrows" ] || fail "init produced no .zenrows in arm $1: $(tail -3 "/tmp/init-$1.log")"
65
65
case"$1"in
66
-
pointer) cat >"$d/CLAUDE.md"<<'PTR'
67
-
## Zenrows
68
-
69
-
This project uses the Zenrows CLI for protected web data. Prefer it over a plain
70
-
HTTP client or a local browser when a page is behind anti-bot protection.
71
-
72
-
- `zenrows --help` lists every command.
73
-
- `zenrows status --json` reports the live capability matrix.
0 commit comments