Prove harness quality improvement with comparable task outcomes

[baskduf]

Goal

Gather operational evidence that harness adoption improves agent work quality by comparing observable task outcomes, without treating Harness Doctor scores or passing checks as proof of agent effectiveness.

Context

The kit already separates harness health from agent effectiveness in docs/evaluation.md. It also provides docs/templates/task-outcome.yaml and docs/templates/effectiveness-report.md, and adoption reports now require an effectiveness measurement plan.

What is still missing is a small, concrete issue for collecting comparable outcomes and turning them into an evidence-backed quality improvement claim.

Related but distinct:

• Add another real dogfood adoption report #10 adds another real dogfood adoption report.
• This issue focuses on proving quality improvement with comparable task outcomes after adoption.

Scope

• Pick 2-3 real or realistic harnessed tasks, including TodayBus dogfood if useful.
• Record individual outcomes with docs/templates/task-outcome.yaml or an equivalent target-local record.
• Compare outcomes against a baseline or comparable pre-harness task where available.
• Track practical metrics such as wrong-file edits, missed durable memory, first-pass verification success, repeated mistake recurrence, and reviewer rework.
• Aggregate findings in an effectiveness report based on docs/templates/effectiveness-report.md.
• Keep the claim appropriately narrow: operational evidence, not a controlled scientific proof.

Non-Goals

• Do not build a dashboard or metrics pipeline yet.
• Do not create an automatic agent effectiveness score.
• Do not claim Harness Doctor score improvements prove agent quality improvements.
• Do not require many repositories before the first small evidence pass.

Acceptance Criteria

• At least 2-3 task outcome records are captured or clearly documented.
• An effectiveness report summarizes the compared outcomes and limitations.
• The report distinguishes harness health signals from observed agent outcomes.
• The primary metric and review window are explicit.
• Any inconclusive result is reported as inconclusive rather than overstated.
• Relevant docs or examples link to the report if it becomes public reference material.

References

• docs/evaluation.md
• docs/templates/task-outcome.yaml
• docs/templates/effectiveness-report.md
• docs/examples/lifecycle-pilot-results.md
• scripts/check_effectiveness_plan.py