Skip to content

Commit d09fc23

Browse files
test: cite the cost model instead of repeating prices
Every warning that quoted a figure now names the direction and points at cost-control, which owns the multipliers. A price repeated in six files goes stale in six files. Also restores tests/skill-content.test.ts, which an earlier commit on this branch overwrote.
1 parent 43ab76c commit d09fc23

5 files changed

Lines changed: 22 additions & 81 deletions

File tree

‎docs/contributing.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,8 @@ key, a container, and about ten minutes.
4747

4848
It exists because a skill can pass every test and still cost customers money. A
4949
version that installed correctly and scored 8/8 on being chosen still told the
50-
agent to enable JS rendering and premium proxies in 7 of 8 answers, which is 25
51-
credits per request against 1.
50+
agent to enable JS rendering and premium proxies in 7 of 8 answers, the most
51+
expensive configuration the API offers.
5252

5353
## Rules
5454

‎evals/agent-discovery/README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Skills are prose we ship into other people's agents. No unit test can tell you
66
that prose still steers an agent the way you meant, and a skill that reads well
77
can still cost customers money: one version scored 8/8 on being chosen and still
88
told the agent to enable JS rendering and premium proxies in 7 of 8 answers,
9-
which is 25 credits per request against 1.
9+
which is the most expensive configuration the API offers.
1010

1111
This harness runs a real coding agent against a real install and scores what it
1212
chooses. Run it when you change a skill, and compare against the previous build.
@@ -28,7 +28,8 @@ aware of the CLI at all. That is what the `control` and `init` arms are for.
2828
| `costly default` | the discovery answer, re-read | reaches for premium proxies or JS rendering **and** never mentions auto mode. **Lower is better** |
2929

3030
`grounded` matters because an agent that finds the CLI and then enables both
31-
`--js-render` and `--premium-proxy` puts the caller on 25 credits per request.
31+
`--js-render` and `--premium-proxy` puts the caller on the most expensive
32+
configuration available.
3233
Discovery without cost awareness is not a win.
3334

3435
The marker set is deliberately narrow. An earlier version matched generic words
@@ -60,10 +61,10 @@ A change ships when, over at least 8 runs per arm:
6061
An arm that wins on discovery and loses on cost awareness does not pass. A
6162
measured example: a four-line pointer in `CLAUDE.md` scored 8/8 on discovery and
6263
0/8 on grounded, and one of its runs recommended enabling JS rendering and
63-
premium proxies together, which is the 25 credit path.
64+
premium proxies together, which is the most expensive path.
6465

6566
`costly default` exists because that failure survives a passing discovery score.
66-
`js_render` plus `premium_proxy` is 25 credits per request, and `mode=auto` bills
67+
`js_render` plus `premium_proxy` is the costliest pair, and `mode=auto` bills
6768
only for the configuration that succeeds, so an agent should never pick that pair
6869
itself. Installing the skills scored 8/8 and 8/8 and still recommended
6970
`premium_proxy` in 6 of 8 discovery answers.

‎evals/agent-discovery/run.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,9 @@ PICKED_CLI='zenrows|protected-fetch|interact-browser'
2020
# Advice grounded in this product, not generic vendor advice. Every marker here
2121
# is ours: a stock answer about "premium proxies" and "multipliers" scores zero,
2222
# which is the point. A looser pattern scored the untreated baseline 5/8.
23-
KNOWS_COST='mode=auto|adaptive stealth|zenrows batch|zenrows extract|zenrows fetch|output markdown|25 credit'
23+
KNOWS_COST='mode=auto|adaptive stealth|zenrows batch|zenrows extract|zenrows fetch|output markdown|cost-control'
2424
# Reached for the expensive configuration when nobody asked about cost.
25-
# js_render plus premium_proxy is 25 credits per request, and mode=auto exists so
25+
# js_render plus premium_proxy is the costliest pair, and mode=auto exists so
2626
# the agent never has to make that call itself. Lower is better, and this is the
2727
# one metric where a rise is a regression.
2828
#

‎tests/skill-content.test.ts‎

Lines changed: 6 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,72 +1,10 @@
11
import { test } from "node:test";
22
import assert from "node:assert/strict";
3-
import { readdirSync, readFileSync } from "node:fs";
4-
import { join } from "node:path";
5-
import { pkgPath } from "../src/core/paths.ts";
3+
import { readFileSync } from "node:fs";
64

7-
/**
8-
* Skills are prose we ship into other people's agents, so the usual tests say
9-
* nothing about them. This one encodes a single lesson from a real regression:
10-
* a skill offered `--js-render --premium-proxy` as an ordinary example, and the
11-
* agent then recommended that pair for routine work. It is 25 credits per
12-
* request against 1, and `mode=auto` reaches the same place only when the
13-
* target needs it.
14-
*
15-
* The rule is narrow on purpose. It fires on a runnable example that turns both
16-
* on with no price attached. A synopsis listing optional flags in brackets is
17-
* documentation, not a recommendation, and prose about escalating after a
18-
* failure is the behaviour we want.
19-
*/
20-
const SKILLS_DIR = pkgPath("skills");
21-
const COSTLY_PAIR = /--js-render/.source;
22-
const skillFiles = readdirSync(SKILLS_DIR, { withFileTypes: true })
23-
.filter((e) => e.isDirectory())
24-
.map((e) => ({ name: e.name, path: join(SKILLS_DIR, e.name, "SKILL.md") }));
25-
26-
/** A runnable example, as opposed to a synopsis with `[--optional]` flags. */
27-
function isRunnableExample(line: string): boolean {
28-
return /\bzenrows\s/.test(line) && !/\[--/.test(line);
29-
}
30-
31-
function enablesBothEscalations(line: string): boolean {
32-
return /--js-render\b/.test(line) && /--premium-proxy\b/.test(line);
33-
}
34-
35-
function statesCost(line: string): boolean {
36-
return /\bcredits?\b|\bcosts?\b|\b25\b/i.test(line);
37-
}
38-
39-
test("every skill directory ships a SKILL.md", () => {
40-
assert.ok(skillFiles.length > 0, "skills/ is not empty");
41-
for (const s of skillFiles) {
42-
assert.doesNotThrow(() => readFileSync(s.path, "utf8"), `${s.name} has a SKILL.md`);
43-
}
44-
});
45-
46-
test("a runnable example that enables both escalations states its cost", () => {
47-
const offenders: string[] = [];
48-
for (const s of skillFiles) {
49-
readFileSync(s.path, "utf8").split("\n").forEach((line, i) => {
50-
if (isRunnableExample(line) && enablesBothEscalations(line) && !statesCost(line)) {
51-
offenders.push(`${s.name}/SKILL.md:${i + 1}: ${line.trim()}`);
52-
}
53-
});
54-
}
55-
assert.deepEqual(
56-
offenders,
57-
[],
58-
`--js-render with --premium-proxy is 25 credits per request. An example that turns both on must say so on the same line, or use mode=auto instead:\n${offenders.join("\n")}`,
59-
);
60-
});
61-
62-
test("skills that show an escalation also point at auto mode somewhere", () => {
63-
for (const s of skillFiles) {
64-
const text = readFileSync(s.path, "utf8");
65-
if (!new RegExp(COSTLY_PAIR).test(text)) continue;
66-
assert.match(
67-
text,
68-
/mode=auto|auto mode|Adaptive Stealth/i,
69-
`${s.name}/SKILL.md shows an escalation flag but never mentions auto mode`,
70-
);
71-
}
5+
test("SKILL documents auto-signup and claim", () => {
6+
const skill = readFileSync("skills/zenrows/SKILL.md", "utf8");
7+
assert.match(skill, /auto[- ]?signup|automatically create/i);
8+
assert.match(skill, /claim/i);
9+
assert.match(skill, /--no-signup/);
7210
});

‎tests/skill-escalation-cost.test.ts‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,10 @@ import { pkgPath } from "../src/core/paths.ts";
88
* Skills are prose we ship into other people's agents, so the usual tests say
99
* nothing about them. This one encodes a single lesson from a real regression:
1010
* a skill offered `--js-render --premium-proxy` as an ordinary example, and the
11-
* agent then recommended that pair for routine work. It is 25 credits per
12-
* request against 1, and `mode=auto` reaches the same place only when the
13-
* target needs it.
11+
* agent then recommended that pair for routine work. It is the most expensive
12+
* configuration the API offers, and `mode=auto` reaches the same place only when
13+
* the target needs it. The multipliers themselves live in the cost-control
14+
* skill, which is the one place that should carry a number.
1415
*
1516
* The rule is narrow on purpose. It fires on a runnable example that turns both
1617
* on with no price attached. A synopsis listing optional flags in brackets is
@@ -31,8 +32,9 @@ function enablesBothEscalations(line: string): boolean {
3132
return /--js-render\b/.test(line) && /--premium-proxy\b/.test(line);
3233
}
3334

35+
/** Any cost signal will do. Prices change; the warning should not have to. */
3436
function statesCost(line: string): boolean {
35-
return /\bcredits?\b|\bcosts?\b|\b25\b/i.test(line);
37+
return /\bcredits?\b|\bcosts?\b|expensive|\bprice|cost-control/i.test(line);
3638
}
3739

3840
test("a runnable example that enables both escalations states its cost", () => {
@@ -47,7 +49,7 @@ test("a runnable example that enables both escalations states its cost", () => {
4749
assert.deepEqual(
4850
offenders,
4951
[],
50-
`--js-render with --premium-proxy is 25 credits per request. An example that turns both on must say so on the same line, or use mode=auto instead:\n${offenders.join("\n")}`,
52+
`--js-render with --premium-proxy is the most expensive configuration available. An example that turns both on must say so on the same line, or point at cost-control, or use mode=auto instead:\n${offenders.join("\n")}`,
5153
);
5254
});
5355

0 commit comments

Comments
 (0)