diff --git a/README.md b/README.md index 23d6b833..44bd2e27 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,10 @@ npm install -g cue-ai --- +

+ cue picking a cuecard and launching the agent +

+ ## Why this exists If you've been using AI coding agents for a while, you've probably collected a pile of skills, MCP servers, and custom instructions. Maybe hundreds. Here's the problem: @@ -57,6 +61,10 @@ cue fixes this by scoping everything per directory. Your Medusa shop loads the M ### Before vs after — in numbers +

+ Everything-loaded vs a scoped cuecard — always-on context compared +

+ | Loadout | Always-on context | Cost / 100 msgs (Sonnet input) | |---|---|---| | Everything loaded (`full` profile) | ~81k tokens | ~$24 | @@ -93,6 +101,10 @@ One cuecard per project. Your agent reads the right one the moment you launch it No daemon, no background process. cue intercepts the *call* to your agent, resolves the directory's cuecard, materializes it once, then hands off to the real binary: +

+ cue resolve to materialize to exec flow +

+ ``` you type `claude` │ @@ -249,6 +261,10 @@ cue failures --propose # let Claude draft profile improvements from failur `cue --help` shows the full ~50-subcommand surface; the set above covers a typical week. +

+ cue optimizer dashboard — skills, MCPs, CLIs, and usage per profile +

+ --- ## API diff --git a/evals/agent-eval/.env.example b/evals/agent-eval/.env.example new file mode 100644 index 00000000..1ca5144a --- /dev/null +++ b/evals/agent-eval/.env.example @@ -0,0 +1,16 @@ +# cue-agent-eval — credentials +# +# Pick ONE agent API key. The experiments default to `agent: 'claude-code'` +# (direct Anthropic), so ANTHROPIC_API_KEY is the simplest path. +ANTHROPIC_API_KEY=sk-ant-... + +# Or, to route through the Vercel AI Gateway instead, set this and change the +# experiments to `agent: 'vercel-ai-gateway/claude-code'`. This key also enables +# the optional failure classifier (infra/timeout vs model failures). +# AI_GATEWAY_API_KEY=your-ai-gateway-api-key + +# Sandbox: the experiments use `sandbox: 'docker'`, so NO Vercel token is needed. +# (Docker must be installed and running.) To use the Vercel sandbox instead, +# set one of these and drop `sandbox: 'docker'` from the experiment configs. +# VERCEL_TOKEN=your-vercel-token +# VERCEL_OIDC_TOKEN=your-oidc-token diff --git a/evals/agent-eval/.gitignore b/evals/agent-eval/.gitignore new file mode 100644 index 00000000..43a3bf9b --- /dev/null +++ b/evals/agent-eval/.gitignore @@ -0,0 +1,7 @@ +node_modules/ +dist/ +.env +.env.local +results/ +*.log +.DS_Store diff --git a/evals/agent-eval/README.md b/evals/agent-eval/README.md new file mode 100644 index 00000000..cd3cfc86 --- /dev/null +++ b/evals/agent-eval/README.md @@ -0,0 +1,72 @@ +# cue-agent-eval + +A/B test cue **profiles** with [`@vercel/agent-eval`](https://github.com/vercel-labs/agent-eval). +Same task, same model, run in an isolated sandbox — the **only variable is the +cue profile's `CLAUDE.md`** (persona + rules + skill-routing). The output is a +**pass rate** per profile, so "gstack is better than core for X" stops being a +vibe and becomes a number. + +``` +experiments/core.ts baseline +experiments/gstack.ts 58-role full profile +experiments/improver.ts goal-with-a-check loop +evals/create-button/ one starter task (PROMPT.md + EVAL.ts) +lib/with-profile.ts injects a profile's CLAUDE.md into the sandbox +``` + +## How a profile becomes the variable + +`lib/with-profile.ts` adds a `setup()` that runs **on the host**: + +1. `cue launch --rematerialize` — materializes the profile's full + runtime (persona + rules + skill-routing + `_always` fragments → `CLAUDE.md`) + **without exec'ing claude**, and prints `{ runtimeDir }`. +2. Reads that `CLAUDE.md` and writes it into the sandbox project root, so the + in-sandbox Claude Code runs under that profile's instructions. + +**Not injected** (by design): MCP servers, hooks, and the headroom proxy env. +They need keys/network and would fail to start in a throwaway sandbox. The +measured variable is the profile's *instructions*, not its infra. Skill **bodies** +aren't injected yet either (the `CLAUDE.md` already carries each profile's skill +list + routing) — see the extension note at the bottom of `lib/with-profile.ts`. + +## Prerequisites + +- `@vercel/agent-eval` — installed globally (`agent-eval --help`). +- **Docker** — the experiments use `sandbox: 'docker'` (no Vercel token needed). +- **An agent API key** — `ANTHROPIC_API_KEY` (direct, the default) **or** + `AI_GATEWAY_API_KEY` (gateway). Without one the sandbox can't authenticate, so + runs fail. `--dry` needs no key. +- `cue` on `PATH` (this repo's CLI) — `with-profile.ts` shells out to it. + Run the eval from a **plain shell**, not from inside a cue-launched session: + `CUE_LAUNCHING=1` trips cue's recursion guard and `--rematerialize` would fail. + +## Run + +```bash +cd evals/agent-eval +cp .env.example .env # add ANTHROPIC_API_KEY (or AI_GATEWAY_API_KEY) +npm install # only needed to typecheck / run EVAL.ts locally + +npx @vercel/agent-eval --dry # preview all 3 profiles — no API calls, no cost +npx @vercel/agent-eval # run all 3 (core, gstack, improver) +npx @vercel/agent-eval core # run just one +npx @vercel/agent-eval --smoke # one run per experiment — verify keys/sandbox +``` + +Results land in `results///`. Compare `passRate` in each +`summary.json` across the three profiles. + +## Extend + +- **More tasks:** add `evals//` with `PROMPT.md` + `EVAL.ts` (+ `package.json`, + `src/`). They run against every experiment automatically. +- **More profiles:** copy an experiment file and change `withProfile('')`. +- **All profiles:** map `withProfile` over `cue list` output from a generator + script (heavier; most configs you'd never run — start with the 3 here). + +## Cost note + +Each run spins up a sandbox, installs Claude Code, and runs the agent. `runs: 3` +× 3 profiles × tasks = real API tokens. Start with `--dry`, then `--smoke`, then +a full run on a small task set. diff --git a/evals/agent-eval/evals/create-button/EVAL.ts b/evals/agent-eval/evals/create-button/EVAL.ts new file mode 100644 index 00000000..2eabbb27 --- /dev/null +++ b/evals/agent-eval/evals/create-button/EVAL.ts @@ -0,0 +1,23 @@ +import { readFileSync, existsSync } from 'fs'; +import { execSync } from 'child_process'; +import { test, expect } from 'vitest'; + +test('Button component file exists', () => { + expect(existsSync('src/Button.tsx')).toBe(true); +}); + +test('Button accepts label and onClick props', () => { + const src = readFileSync('src/Button.tsx', 'utf-8'); + expect(src).toContain('label'); + expect(src).toContain('onClick'); +}); + +test('index re-exports Button', () => { + const idx = readFileSync('src/index.ts', 'utf-8'); + expect(idx).toMatch(/Button/); +}); + +test('project builds', () => { + // Throws (and fails the test) if the build fails. + execSync('npm run build', { stdio: 'pipe' }); +}); diff --git a/evals/agent-eval/evals/create-button/PROMPT.md b/evals/agent-eval/evals/create-button/PROMPT.md new file mode 100644 index 00000000..740180ec --- /dev/null +++ b/evals/agent-eval/evals/create-button/PROMPT.md @@ -0,0 +1,8 @@ +Create a reusable Button component. + +Requirements: +- Export a `Button` component from `src/Button.tsx` +- It accepts two props: `label` (string) and `onClick` (() => void) +- It renders a `