[Eval]: ADR Automation Evaluation

[mjunaidca]

This three-part ADR test should:

1. Suggest ADRs for architecturally significant decisions (high impact, multiple alternatives, cross-cutting)
2. NOT suggest ADRs for trivial choices (standard practices, no alternatives, local scope)
3. Cluster related technologies into single ADRs (e.g., "Frontend Stack" not 3 separate ADRs)
4. Wait for user consent before creating (never auto-create ADRs)

Current State:

• ✅ Three-part test documented in AGENTS.md
• ✅ Clustering guidance documented in templates and docs
• ❌ No validation that agents follow these rules
• ❌ Unknown accuracy rates (false positives/negatives)

Current State

What Works

• ✅ ADR template has clustering guidance
• ✅ /adr command has ✅ GOOD vs ❌ BAD examples
• ✅ Documentation explains clustering principle
• ✅ AGENTS.md specifies "wait for consent"

What's Unknown

• ❓ Do agents actually apply three-part test?
• ❓ Do agents cluster or create separate ADRs?
• ❓ Do agents wait for consent or auto-create?
• ❓ False positive rate (suggesting ADR for trivial choices)
• ❓ False negative rate (missing significant decisions)

Success Metrics

Metric	Target	Current	Status
Accuracy
True positives (suggest when should)	≥90%	Unknown	❓
True negatives (don't suggest when shouldn't)	≥95%	Unknown	❓
Clustering
Correct clustering (1 ADR for stacks)	≥90%	Unknown	❓
Avoid over-granular (no 3+ ADRs for single stack)	100%	Unknown	❓
User Consent
Wait for consent (no auto-create)	100%	Unknown	❓
Clear suggestion format	100%	Unknown	❓

Test Scenarios

Part 1: Significance Test Accuracy

Scenario 1.1: Should Suggest ADR (Framework Choice)

context: Planning phase for new project
input: "Let's use Next.js 14 with App Router for the frontend"
analysis:
  impact: High (framework affects entire frontend)
  alternatives: Yes (Remix, Astro, SvelteKit, vanilla React)
  scope: Cross-cutting (affects routing, data fetching, deployment)
expected: 
  suggest_adr: true
  reason: "Passes all 3 parts of significance test"

Scenario 1.2: Should Suggest ADR (Data Architecture)

context: Planning database schema
input: "We'll use PostgreSQL with row-level security for multi-tenancy"
analysis:
  impact: High (affects data model, security, scalability)
  alternatives: Yes (MongoDB, MySQL, Supabase, separate DBs per tenant)
  scope: Cross-cutting (affects auth, queries, migrations)
expected:
  suggest_adr: true
  reason: "Passes all 3 parts of significance test"

Scenario 1.3: Should NOT Suggest ADR (Linter Choice)

context: Setting up project tooling
input: "Install ESLint with Airbnb config"
analysis:
  impact: Low (code quality, not architecture)
  alternatives: No (ESLint is industry standard)
  scope: Local (only affects development)
expected:
  suggest_adr: false
  reason: "Fails alternatives test (standard practice)"

Scenario 1.4: Should NOT Suggest ADR (Utility Library)

context: Implementing date formatting
input: "Use date-fns for date handling"
analysis:
  impact: Low (can be swapped easily)
  alternatives: Yes (moment, dayjs, Intl) but low-stakes
  scope: Local (isolated to date utilities)
expected:
  suggest_adr: false
  reason: "Fails impact and scope tests"

Part 2: Clustering Accuracy

Scenario 2.1: Should Cluster (Frontend Stack)

context: Planning frontend architecture
input: "Use Next.js 14, Tailwind CSS v3, deploy on Vercel"
expected:
  adr_count: 1
  adr_title: "Frontend Technology Stack"
  components:
    - "Framework: Next.js 14 (App Router)"
    - "Styling: Tailwind CSS v3"
    - "Deployment: Vercel"
    - "State: React Context (start simple)"
reason: "These work together as integrated solution"

Scenario 2.2: Should Cluster (Auth Stack)

context: Planning authentication
input: "Use NextAuth.js with GitHub OAuth and PostgreSQL session storage"
expected:
  adr_count: 1
  adr_title: "Authentication Architecture"
  components:
    - "Auth Library: NextAuth.js v5"
    - "OAuth Provider: GitHub"
    - "Session Storage: PostgreSQL"
    - "Token Strategy: JWT"
reason: "Auth components form cohesive system"

Scenario 2.3: Should Separate (Independent Stacks)

context: Planning full-stack architecture
input: "Next.js frontend on Vercel, Node.js API on Railway, PostgreSQL on Supabase"
expected:
  adr_count: 3
  adr_titles:
    - "Frontend Technology Stack"
    - "Backend API Architecture"
    - "Database and Storage"
reason: "Independent choices, could change separately"

Scenario 2.4: Should NOT Over-Cluster

context: Planning project tooling and architecture
input: "Next.js frontend, PostgreSQL database, GitHub Actions CI, ESLint linting"
expected:
  adr_count: 2
  adr_titles:
    - "Frontend Technology Stack" (Next.js)
    - "Data Architecture" (PostgreSQL)
  not_included:
    - GitHub Actions (standard practice)
    - ESLint (standard practice)
reason: "Don't cluster unrelated decisions or non-ADR-worthy choices"

Part 3: User Consent Workflow

Scenario 3.1: Proper Suggestion Format

context: After /plan command with framework decision
expected_message: |
  📋 Architecture decisions identified. Document reasoning and tradeoffs?
  Run `/adr Frontend Technology Stack` to create ADR.
  
  Components to document:
  - Framework: Next.js 14
  - Styling: Tailwind CSS
  - Deployment: Vercel
  
user_action: Must explicitly run /adr command
auto_create: false

Scenario 3.2: No Suggestion for Trivial Choices

context: After discussing linter setup
expected_message: null
reason: "ESLint is standard practice, doesn't pass significance test"
auto_create: false

Agents to Test

• GitHub Copilot
• Claude Code
• Cursor
• Gemini CLI
• Qwen Code CLI

Evaluation Rubric

For each scenario, score agent on:

• Accuracy (0-1): Correct suggestion decision
• Clustering (0-1): Correct grouping
• Consent (0-1): Waited for user confirmation
• Quality (0-1): Clear, helpful suggestion message

Overall Score: Average across all scenarios

Blocking Priority

Yes - critical for v0

ADR automation is core value proposition. We need data on:

1. Whether three-part test prevents ADR spam
2. Whether clustering prevents over-granular documentation
3. Whether consent workflow balances automation with control

Related Issues/PRs

• Related to protocol-templates/AGENTS.md (three-part test definition)
• Related to templates/adr-template.md (clustering guidance)
• Related to templates/commands/adr.md (command implementation)
• Related to docs-plus/05_spec_driven_development/05_adr/ (ADR documentation)
• Related to Issue Qwen Code CLI: Hands Advanced Modules (MCP, Subagents, Evals, Checkpointing, Vision) #1 (PHR Coverage) - similar eval methodology
• Related to Issue Fix header formatting in the Step-by-Step Guide to Making a Pull Requ… #2 (AGENTS.md Compliance) - validates protocol effectiveness