Skip to content

eval-calls.md

Latest commit

 

History

History
192 lines (159 loc) · 10.5 KB

eval-calls.md

File metadata and controls

192 lines (159 loc) · 10.5 KB
Raw

Eval call plans: pricing a paid gate without running it

A paid gate (a benchmark, an eval smoke, any command that calls a model provider for money) is priced before anyone approves it. gstack does not run the gate, or any other repository command, to learn its cost. The repository commits a call-plan file: plain data that names the exact gate command, the commit and inputs it was computed from, and one line per call purpose with its model and token counts. gstack-eval-plan price-gate reads that file, checks that it still describes what is being approved, prices it from lib/pricing.ts, and writes a receipt. Plan: field wave CD2 (docs/designs/GSTACK_FIELD_WAVE_2026_10_12.md, gate decision d5b as amended).

gstack-eval-plan price-gate --gate-file <file> --cap-usd 20 [--smoke-cap-usd 2] [--plan-file <file>]

--gate-file reads the exact gate command from a file (one trailing newline is dropped), so a command with quotes or $ never passes through a shell string; --gate "<command>" takes it inline. Give one of the two.

Where the file comes from

  1. --plan-file <file>: an operator-given file, read from disk.
  2. Otherwise price_file in .gstack/pregate.json as committed on origin/<base> (--base, default origin/HEAD's branch): a repo-relative path whose bytes are read from the HEAD commit (git show HEAD:<path>), never from the working tree.

Nothing in the file is executed: not the gate command, not a precondition, not an input path. The file is parsed as text.

File format, wire version v1

# gstack eval call plan, wire format v1 (docs/eval-calls.md); priced by `gstack-eval-plan price-gate`, never executed
GSTACK_EVAL_GATE v1: bun run eval:synopsis --questions 50 --arms opus,sol
GSTACK_EVAL_HEAD v1: 0123456789abcdef0123456789abcdef01234567
GSTACK_EVAL_INPUT v1: sha256=<64 hex> path=evals/synopsis/questions.jsonl
GSTACK_EVAL_PRECONDITION: ANTHROPIC_API_KEY and OPENAI_API_KEY set at run time
GSTACK_EVAL_CALLS v1: purpose=answer model=anthropic/claude-opus-5-5 calls=100 in_tokens=400000 out_tokens=200000 max_out_tokens=8192 scope=full
GSTACK_EVAL_CALLS v1: purpose=judge model=gpt-6.1-sol calls=100 in_tokens=300000 out_tokens=50000 max_out_tokens=2048 scope=full
GSTACK_EVAL_CALLS v1 end: lines=2
Line Count Meaning
GSTACK_EVAL_GATE v1: <command> exactly 1 the exact gate command this plan prices; compared to --gate by string equality
GSTACK_EVAL_HEAD v1: <sha> exactly 1 the full commit sha the plan was computed at
GSTACK_EVAL_INPUT v1: sha256=<hex> path=<path> 0 or more each input the counts came from (rows, datasets, prompts), repo-relative, no ..
GSTACK_EVAL_PRECONDITION: <text> 0 or more what must hold at run time; every one is echoed as GATE_PRECONDITION:
GSTACK_EVAL_CALLS v1: … 0 or more one line per call purpose (unique purpose + model)
GSTACK_EVAL_CALLS v1 end: lines=<n> exactly 1, last n = the number of call lines; a missing or miscounted end line means a partial producer and refuses

Blank lines and lines starting with # are ignored. Every other line must be one of the above; an unknown kind, an unknown version (v2), a missing or extra key, a duplicate key, a non-integer count, calls=0, or any content after the end line refuses with EVAL_DRY_RUN_FAILED, naming the line.

Call line keys

Key Meaning
purpose name of the call purpose ([A-Za-z0-9._:-]+), for example answer, judge, embed
model model id; a provider prefix (anthropic/, openai:, voyage:) is accepted
calls number of calls for this purpose (at least 1)
in_tokens aggregate input tokens across all of this purpose's calls
out_tokens aggregate expected output tokens across all of this purpose's calls
max_out_tokens the per-call output reservation (the max_tokens the call sends)
scope full (the counted run) or smoke (a small pre-run check)

The aggregate totals are what the purpose is expected to cost: usd = price(in_tokens, out_tokens). The per-call reservation is what one call can cost at most, which is what a provider or a spend ledger reserves: reservation_usd = price(ceil(in_tokens / calls), max_out_tokens).

Binding checks (EVAL_PLAN_STALE)

The plan refuses, with no receipt, when it no longer describes what is being approved:

  • the GSTACK_EVAL_GATE command differs from --gate by even one character;
  • the GSTACK_EVAL_HEAD commit is not HEAD and any file other than the plan file itself changed between them (a committed plan file sits one commit on top of the head it was computed at, so its own commit is allowed);
  • an input's sha256 of its bytes in the HEAD commit differs from its GSTACK_EVAL_INPUT line, or the input is not in HEAD;
  • the base policy declares a price_file that is not in the HEAD commit.

--allow-unpriced never overrides these: only a gate with no plan declared at all, or a plan with zero call lines, is unpriced.

Pricing outcomes

GATE_PLAN: file=evals/gate.calls source=price_file sha256=<hex> head=<sha12> inputs=1 verified
GATE_PRECONDITION: ANTHROPIC_API_KEY and OPENAI_API_KEY set at run time
GATE_CALLS: purpose=answer model=anthropic/claude-opus-5-5 calls=100 usd=5.6 reservation_usd=0.17984 scope=full
GATE_CALLS: purpose=judge model=gpt-6.1-sol calls=100 usd=1.1 reservation_usd=0.02648 scope=full
CHECK: reservation-fits-cap ok (largest one-call reservation $0.1798)
GATE_PRICE: total=6.7 calls=200 purposes=answer,judge receipt=.gstack/tmp/price-gate/price-gate-<gate12>-<head12>.json
Outcome Exit Code
priced, under the caps 0 —
a model with no row in lib/pricing.ts: that row and the total print unknown, never zero 1 EVAL_PRICE_MISSING
total above --cap-usd 1 EVAL_GATE_OVER_CAP
one call's reservation above --cap-usd, or above --smoke-cap-usd for a scope=smoke row (CHECK: reservation-fits-cap exceeds (…)) 1 EVAL_GATE_OVER_CAP
no plan declared (no --plan-file and no price_file key) or a file with zero call lines 1 EVAL_GATE_UNPRICED
the same with --allow-unpriced "<reason>" 0 prints GATE_UNPRICED_OVERRIDE: <reason> (owner-authorized override; not authorization to spend) and records it
malformed or partial file 1 EVAL_DRY_RUN_FAILED
stale binding, or a declared price_file missing from HEAD (not overridable) 1 EVAL_PLAN_STALE
origin/<base> unresolvable while reading price_file 3 POLICY_SOURCE_UNAVAILABLE
.gstack/pregate.json on the base is invalid 3 PREGATE_CONFIG_INVALID
unknown flag or missing --gate 2 usage

The price is an estimate: the table prices standard text tokens and excludes the billing categories listed in lib/pricing.ts (cache writes, tool fees, service tiers and long-context surcharges, media tokens, seat-metered runs).

Receipt

Every outcome past the binding checks writes a JSON receipt to --out (default <repo>/.gstack/tmp/price-gate/), named price-gate-<gate sha256 12>-<head 12>.json, and prints its path on the GATE_PRICE: line. --json prints the receipt (plus receipt_path) instead of the lines.

Field Meaning
schema_version 1
kind gstack-eval-price-gate
created_at, gstack_revision when, and which gstack (VERSION and commit) priced it
gate_command, gate_sha256 the approved command and its sha256
head_sha HEAD when priced
plan file, source (flag, price_file, none), sha256 of the plan file bytes, declared_head
inputs each { path, sha256 }, verified at HEAD
preconditions every precondition line
rows each call line plus usd and reservation_usd (null when unpriced)
total_usd, calls, purposes the totals; total_usd is null when unknown
cap_usd, smoke_cap_usd the caps passed, or null
result, code priced, price_missing, over_cap, unpriced, unpriced_override, and the result code
unpriced_reason, unpriced_override why the gate is unpriced, and the owner's override reason
excluded_billing the billing categories the estimate leaves out

Reference emitter

A consuming repo writes the file from the same code that builds its calls, so the counts cannot drift from the run. formatEvalCalls in lib/eval-plan.ts is the reference; copy it:

import { createHash } from 'node:crypto';
import { execFileSync } from 'node:child_process';
import { readFileSync } from 'node:fs';

const KEYS = ['purpose', 'model', 'calls', 'in_tokens', 'out_tokens', 'max_out_tokens', 'scope'] as const;
type Row = { purpose: string; model: string; calls: number; in_tokens: number; out_tokens: number; max_out_tokens: number; scope: 'full' | 'smoke' };

export function formatEvalCalls(p: { gate: string; head: string; inputs: { path: string; sha256: string }[]; preconditions: string[]; rows: Row[] }): string {
  return [
    '# gstack eval call plan, wire format v1 (docs/eval-calls.md); priced by `gstack-eval-plan price-gate`, never executed',
    `GSTACK_EVAL_GATE v1: ${p.gate}`,
    `GSTACK_EVAL_HEAD v1: ${p.head}`,
    ...p.inputs.map(i => `GSTACK_EVAL_INPUT v1: sha256=${i.sha256} path=${i.path}`),
    ...p.preconditions.map(t => `GSTACK_EVAL_PRECONDITION: ${t}`),
    ...p.rows.map(r => `GSTACK_EVAL_CALLS v1: ${KEYS.map(k => `${k}=${r[k]}`).join(' ')}`),
    `GSTACK_EVAL_CALLS v1 end: lines=${p.rows.length}`,
  ].join('\n') + '\n';
}

// Count tokens the way the run will, without calling a provider, then:
const head = execFileSync('git', ['rev-parse', 'HEAD'], { encoding: 'utf8' }).trim();
const input = 'evals/synopsis/questions.jsonl';
const sha256 = createHash('sha256').update(readFileSync(input)).digest('hex');
// writeFileSync('evals/gate.calls', formatEvalCalls({ gate, head, inputs: [{ path: input, sha256 }], preconditions, rows }));

Hash the inputs as committed (the file at HEAD), produce the plan at a clean HEAD, commit the plan file on its own commit, and set "price_file": "evals/gate.calls" in .gstack/pregate.json.

Related