[EVAL]: Command Template Baseline Evaluation (Project-Based)

[mjunaidca]

All command templates should:

1. Execute successfully when given appropriate context (≥95% success rate)
2. Follow template structure (output matches expected format)
3. Use project context (reference constitution, spec, previous outputs)
4. Produce consistent results (similar input → similar output across runs)
5. Work across all agents (cross-agent consistency ≥80%)

Current State:

• ✅ Command templates exist for all workflow stages
• ✅ Templates have clear structure and prompts
• ❌ No baseline metrics for success rate per command
• ❌ No quality assessment framework
• ❌ No cross-agent consistency data
• ❌ Unknown which commands work best/worst with which agents

Current State

Commands to Test

Command	Stage	Template	Status
/constitution	Initial	constitution.md	❓ Untested
/spec	Specification	spec.md	❓ Untested
/plan	Planning	plan.md	❓ Untested
/architect	Design	architect.md	❓ Untested
/green	TDD Green	green.md	❓ Untested
/red	TDD Red	red.md	❓ Untested
/refactor	TDD Refactor	refactor.md	❓ Untested
/explainer	Documentation	explainer.md	❓ Untested
/phr	Reflection	phr.md	⚠️ Partially tested
/adr	Decisions	adr.md	⚠️ Partially tested

What We Know

• ✅ Templates exist and have structure
• ✅ Commands work in manual testing
• ⚠️ Some informal experience with success/failure

What We Don't Know

• ❓ Success rate per command per agent
• ❓ Output quality consistency
• ❓ Context awareness level
• ❓ Which commands fail most often
• ❓ Common failure patterns
• ❓ Cross-agent consistency

Success Metrics

Per-Command Metrics

Command	Success Rate	Quality Score	Context Use	Consistency	Target
/constitution	❓	❓	❓	❓	≥95%
/spec	❓	❓	❓	❓	≥95%
/plan	❓	❓	❓	❓	≥90%
/architect	❓	❓	❓	❓	≥90%
/green	❓	❓	❓	❓	≥85%
/red	❓	❓	❓	❓	≥85%
/refactor	❓	❓	❓	❓	≥85%
/explainer	❓	❓	❓	❓	≥90%
/phr	❓	❓	❓	❓	≥95%
/adr	❓	❓	❓	❓	≥90%

Cross-Agent Metrics

Agent	Avg Success	Avg Quality	Best Commands	Worst Commands	Overall
Copilot	❓	❓	❓	❓	❓
Claude	❓	❓	❓	❓	❓
Cursor	❓	❓	❓	❓	❓
Gemini	❓	❓	❓	❓	❓
Cline	❓	❓	❓	❓	❓

Target: ≥85% overall success rate across all commands and agents