[Eval]: PHR Coverage Evaluation Framework

[mjunaidca]

PHRs should be created for ~95% of work sessions, not just / commands. AGENTS.md specifies "After completing ANY work, automatically create PHR" but currently PHRs only fire for formal SDD commands due to command-rules.md injection.

Current State

✅ Working: PHRs created for / commands (command-rules.md injected into templates)
❌ Not Working: PHRs for general conversations, generate prompts, or ad-hoc discussions
❌ Unknown: Actual PHR creation rate across all interaction types

Root Cause:

• AGENTS.md contains broad principle ("capture ALL work")
• command-rules.md injection only applies to formal commands
• No enforcement mechanism for general agent conversations

Success Metrics

Metric	Target	Current	Status
PHR creation rate	≥95% of sessions	Unknown	❓
Context completeness	100% (no truncation)	Unknown	❓
Correct location	100% (docs/prompts/)	~100% for / commands	⚠️
Correct stage detection	≥90% accuracy	Unknown	❓

Test Scenarios

Scenario 1: Formal Commands (Current Working State)

Input: /plan Design authentication system
Expected: PHR created in docs/prompts/architect/
Status: ✅ WORKING (command-rules.md injection)

Scenario 2: General Conversations

Input: "Should we use GraphQL or REST for the API?"
Expected: PHR created in docs/prompts/general/
Status: ❌ NOT WORKING (AGENTS.md not enforced)

Scenario 3: Generate Prompts

Input: "Generate a React login component"
Expected: PHR created in docs/prompts/green/ (implementation stage)
Status: ❌ NOT WORKING (AGENTS.md not enforced)

Scenario 4: Architecture Discussions

Input: "Let's discuss database schema for users and posts"
Expected: PHR created in docs/prompts/architect/
Status: ❌ NOT WORKING (AGENTS.md not enforced)

Scenario 5: Quick Questions

Input: "What's the API endpoint for user login?"
Expected: PHR created in docs/prompts/general/
Status: ❌ NOT WORKING (AGENTS.md not enforced)

Agents to Test

• Claude Code
• Cursor
• Gemini CLI
• Qwen Code CLI
• GitHub Copilot

Related Issues/PRs

• Related to protocol-templates/AGENTS.md (knowledge capture protocol)
• Related to memory/command-rules.md (command injection)
• See docs-plus/05_spec_driven_development/ for PHR documentation

Acceptance Criteria

• Baseline metrics collected for all 5 supported agents
• PHR creation rate measured and documented
• Gaps identified and documented
• Recommendations provided for improving coverage
• Report published with findings