feat(eval): Phase 2 — Neutral Annotator Scripts

[kkeeling]

Eval Phase 2: Neutral Annotator Scripts

Implements the four scripts specified in the Phase 2 neutral annotator spec for automated trace annotation.

New Scripts

Script	Purpose
scripts/sample-traces.ts	Query obs DB for unannotated traces, output prioritized work queue JSON
scripts/format-trace.ts	Format a trace as self-contained markdown evaluation document
scripts/eval-rubric.md	5-dimension evaluation rubric with structured JSON output format
scripts/annotate-batch.ts	Main loop: format → evaluate via Claude Sonnet → write annotations

How It Works

# 1. Generate work queue (initial 200 for Phase 2a)
npx tsx scripts/sample-traces.ts --phase2a

# 2. Run annotator batch (processes 20 traces by default)
npx tsx scripts/annotate-batch.ts --batch-size 20

# 3. Or dry-run first
npx tsx scripts/annotate-batch.ts --batch-size 5 --dry-run

Testing

• End-to-end tested: sampler → formatter → annotator → obs API
• 3 traces annotated successfully (1 fail, 2 flag) — visible in dashboard
• Handles compressed traces (gzip from prune stage 1)
• Resumable: tracks processed IDs in work-queue-processed.json

Rubric Dimensions

1. Task Completion — did it do what was asked?
2. Instruction Following — did it follow its system prompt?
3. Tool Use Quality — correct tools, correct arguments?
4. Efficiency — reasonable resource usage?
5. Output Quality — clear, useful, no hallucinations?

Dependencies

• Added @anthropic-ai/sdk for Claude API access
• Uses existing obs API (port 19100) for writing annotations

Next Steps (not in this PR)

• Run Phase 2a: annotate 200 priority traces
• Grace reviews first 20 for rubric calibration
• Set up daily cron for ongoing annotation

Closes MC task jx7f8g8f3fss44ayyd69a21r41833v8h