Research infrastructure for measuring weighted token cost and software quality in realistic coding-agent workflows. Weighted token cost (fresh input + 0.1 × cached input + 6 × output) is the repository's sole token metric; raw provider counters are calculation/audit telemetry only.
The active portfolio is Lifecycle V2 for two medium-project lanes. Both lanes have passed provider-free qualification. Neither lane holds a baseline. Six prompts were rewritten on 2026-08-21 after exploration variance was traced to prompt precision, which mints new qualification and protocol identities, so both lanes need fresh baselines before any comparison. New Codex CLI and OpenCode evaluations use GPT-5.6 Sol/medium; new Claude Code evaluations use direct-Anthropic Claude Opus 5/medium. High-effort conditions are historical only because the added deliberation can increase trajectory divergence. Each task pre-seeds an authentic semantic regression from completed upstream behavior and gives the agent a normal software-engineering objective: implement the requested outcome correctly, search and inspect related code, preserve prior work, and validate appropriately. Prompts state the observable symptom and never name the file, function, or test, so locating the defect remains real retrieval work, and they do not disclose evaluator scoring or controller commands. The controller applies every regression as one composite start before prompt 1; after the final prompt it runs each task verifier and one frozen project-wide compile command. Every task requires affected-component compilation plus one narrow essential-behavior smoke check. Broader tests, behavioral fidelity, style, maintainability, and source review are diagnostics. The active lifecycle portfolio is:
| Sequence | Fixture | Ordered stages |
|---|---|---|
fastify-lifecycle-sequence-v2 |
Fastify | 6 bounded defect repairs |
beets-lifecycle-sequence-v2 |
Beets | 7 bounded defect repairs |
The active registry contains 5 accepted provider-backed sessions: 5 baselines. By sequence: 3 beets-lifecycle-sequence-v2, 2 fastify-lifecycle-sequence-v2. By runtime: Codex CLI 5.
Weighted token cost decomposes as agent steps times weighted cost per step. 58fa838354e5 holds 3 replicates (75, 67, 80 agent steps, spread 19.4%); weighted cost per step spread 4.0%; dc16afea3ad5 holds 2 replicates (69, 73 agent steps, spread 5.8%); weighted cost per step spread 10.0%.
Archived generations: lifecycle-v1-pre-capped-suite-20260815 (2 sessions); lifecycle-v1-pre-corrected-prompts-20260813 (103 sessions); lifecycle-v1-pre-targeted-tests-20260816 (4 sessions); lifecycle-v2-beets-plugin-tasks-20260820 (3 sessions); lifecycle-v2-pre-prompt-repair-20260821 (5 sessions).
Lifecycle V0 was retired on 2026-08-14 under sources/evaluations/audits/lifecycle-v0-framework-retired-20260814.json. Its 212 sessions, their compact artifact roots, and 224 frozen protocols were deleted from the active corpus at the experiment owner's direction.
The retirement is a design judgement, not an allegation about any individual run. V0 prompts used solution-directed task assistance: they prescribed target files, symbols, implementation steps, and validation commands to reduce trajectory variance. That suppresses the repository search and exploration where context-reduction tools actually act, so a V0 token delta is not attributable to a tool's effect on realistic agent work. Provider totals from those runs were real; the workload they measured was not representative. The receipt records that fault separately from the two narrower ones already adjudicated — the 2026-07-18 official-integration parity failures and the Baseline V3 verifier-environment defect corrected by V4.
Papers reporting V0 results are retained and annotated rather than withdrawn, because negative findings and exclusions are part of the research record (ADR 0003). Their numbers are no longer reproducible from this repository. Terraform was a V0-only lane and has no active fixture or task contract.
Start with docs/README.md. The main destinations are:
docs/papers/— completed research papers and phase reports;docs/evaluations/— evaluation design and operator guidance;docs/research/— current roadmap and research direction;docs/tool-dossiers/— tool index and source-inspection dossiers;templates/— blank outlines and reusable templates.
data/workflow-task-sequences.json— Lifecycle V2 contracts.data/repository-fixtures.json— pinned fixture readiness.sources/evaluations/fixtures/— task prompts, start patches, controller acceptance, and generated V2 qualification evidence.data/workflow-sessions.json— active retained provider-backed controls and objective-eligible treatment samples; corrupted treatments are represented only by deletion receipts.docs/evaluations/operations/runbook.md— generated operator runbook.docs/papers/opencode-four-tool-lifecycle-v1-screen-20260808.md— archived pre-correction OpenCode Lifecycle V1 screen.docs/papers/phase-2-lifecycle-v1-natural-use-screening.md— archived pre-correction Lifecycle V1 screening report.sources/evaluations/audits/lifecycle-v0-framework-retired-20260814.json— Lifecycle V0 retirement adjudication.
make check runs the full AGENTS.md required-checks gate — generated-runbook drift, both
contract test suites, repository validation, and a working-tree comparison that fails if the checks themselves changed tracked or untracked state. Run it
before finishing any change to evaluation state; nothing runs it automatically.
make checkIndividual entry points:
python3 scripts/validate_repository.py
python3 scripts/test_workflow_evaluation_contract.py
python3 scripts/test_claude_code_usage_contract.pyBoth require jsonschema (see requirements-dev.txt): every registry record is gated on
schemas/workflow-session-record.schema.json, and validation fails closed without it.
Fixture qualification evidence is executable and generated by:
python3 scripts/generate_workflow_qualification.py fastify-lifecycle-sequence-v2 sources/evaluations/fixtures/medium/fastify-fastify/repo
python3 scripts/generate_workflow_qualification.py beets-lifecycle-sequence-v2 sources/evaluations/fixtures/medium/beetbox-beets/repoExecuted provider-free integration matrices are published with scripts/publish_integration_qualification.py; the publisher rejects nonzero lanes, provider-backed session creation, failed preparation, failed host integration, failed warmup, or failed MCP handshakes.
See AGENTS.md before changing evaluation contracts.