Explore specialist-agent review orchestration for Kodiai

[keithah]

Context

Cloudflare published a useful write-up on operating AI code review at scale: https://blog.cloudflare.com/ai-code-review/

The core idea worth tracking for Kodiai is that AI review should be treated as an orchestration and observability system, not as one large prompt. Their architecture uses risk-tiered review plans, specialist reviewers, a coordinator/reducer, structured outputs, failback, and explicit telemetry.

Why this matters for Kodiai

Recent production work exposed related failure modes in our current single-reviewer path:

• full reviews can exhaust turns before covering enough files;
• prompt/tool availability can drift when the remote workspace changes shape;
• degraded diff paths need explicit truth surfaces;
• retry/bounded-review state needs durable coverage evidence;
• noisy review findings need stronger filtering and traceability.

PR #130 addresses the immediate degraded-diff and handoff-latency issues. This issue tracks the broader architecture follow-up so it does not block the now-path.

Candidate improvements

1. First-class review plan object

   • Represent tier, agents, prompt mode, tool surface, diff source, timeout/max-turn budget, retry policy, and publish policy in one explicit plan.
   • Prevent drift between allowed tools and prompt instructions.

2. Specialist reviewer lanes

   • Start small rather than copying Cloudflare's seven-agent setup.
   • Candidate lanes:
     • correctness/code-quality;
     • security/safety;
     • docs/config/runbook truthfulness.
   • Run specialists only when risk tier or changed paths justify them.

3. Coordinator/reducer pass

   • Specialists emit structured findings.
   • Coordinator dedupes, validates, reclassifies severity, and publishes a single output.
   • Preserve telemetry for kept/dropped/merged findings with reasons.

4. AGENTS/runbook/config freshness reviewer

   • Flag material changes that should update .kodiai.yml, runbooks, prompts, project instructions, or operator docs.
   • This should catch drift when behavior changes but docs/config examples stay stale.

5. User-visible progress/heartbeat for long reviews

   • Surface review phase progress and “still working” signals for long-running reviews.
   • Tie this to existing phase timing and Review Details evidence.

6. Provider/model failback and circuit-breaker policy

   • Classify retryable provider failures separately from auth/context/user aborts.
   • Route around temporary provider failures without treating them as app correctness bugs.

7. Shared context and diff artifact strategy

   • Evaluate storing shared review context and per-file diff artifacts to reduce duplicate prompt tokens.
   • Keep path/symlink/snapshot safety explicit.

Non-goals

• Do not block PR Fix bounded review follow-ups and handoff latency #130 or the current production verification loop.
• Do not introduce seven concurrent reviewers by default.
• Do not auto-block merges based on a new architecture until the signal/noise contract is proven.

Acceptance shape

A future design should define:

• a typed review-plan contract;
• initial specialist lane boundaries;
• structured finding schema;
• coordinator/reducer telemetry;
• rollout gates and cost controls;
• production metrics proving lower latency/noise or better coverage.

[keithah]

Repo-grounded evolution plan: make Kodiai's existing reducer explicit

After reading the current code, the strongest path is narrower than the generic "add agents" plan. Kodiai already has a lot of the verification system in place. The leverage is to turn the implicit orchestration inside src/handlers/review.ts into typed, observable contracts, then add specialist lanes only where the current reducer can prove they improve signal.

Current assets already in the repo:

• src/handlers/review.ts already does the staged review spine: diff analysis, large-PR risk triage, structural-impact lookup, prompt build, Agent SDK execution, finding extraction, enforcement, claim classification, feedback suppression, cluster scoring, guardrail filtering, graph validation, prioritization, Review Details, telemetry, knowledge writes, and continuation handling.
• src/lib/claim-classifier.ts, src/lib/output-filter.ts, src/lib/severity-demoter.ts, and src/lib/guardrail/pipeline.ts are already the beginning of a claim-verification layer.
• src/structural-impact/*, src/review-graph/*, and migrations 033-canonical-code-corpus.sql / 034-review-graph.sql already provide the architecture-aware retrieval substrate.
• src/knowledge/active-rules.ts, migration 035-generated-rules.sql, learning memory, feedback reactions, and suggestion cluster models already point toward review memory and utility calibration.
• src/lib/review-utils.ts already renders Review Details with scope, timing, large-PR boundedness, structural-impact status, token/cost info, suppression counts, and prioritization stats.

So the next step should not be a new parallel architecture. It should be extracting contracts from the current one.

---

Concrete gaps found in the code

1. No first-class ReviewPlan exists.
   Review decisions are spread across review.ts, review-routing.ts, config parsing, large-PR triage, structural-impact setup, prompt-building inputs, timeout handling, and MCP tool selection. This makes it hard to audit what Kodiai intended to do for a run.

2. Findings are recovered from GitHub comments after publication.
   extractFindingsFromReviewComments() pulls structured findings from posted inline comments. That means verification, utility scoring, and knowledge writes happen after the model has already attempted publication. The MCP publish tools are the current output channel, not a clean candidate-finding channel.

3. The reducer exists, but only as inline review-handler logic.
   Enforcement, claim classification, external-claim filtering, guardrails, graph validation, prioritization, and suppression are all useful gates, but they are not represented as a single ReviewReducerResult with kept/dropped/rewritten reasons.

4. review.graphValidation appears to be unreachable from config.
   src/handlers/review.ts checks config.review.graphValidation?.enabled, but src/execution/config.ts does not include graphValidation in reviewSchema. Zod strips unknown fields by default, so .kodiai.yml likely cannot enable this path. This is exactly the kind of prompt/config/tool drift a typed review plan should prevent.

5. Guardrails classify finding titles, not full evidence bundles.
   reviewAdapter.extractClaims() extracts claims from finding.title. That is useful, but the verifier cannot yet reason over a structured body containing claim, evidence, source, verification status, impact, and suggested fix.

6. Review memory exists but is not yet a doctrine contract.
   Active generated rules can be injected, and learning memory/feedback can suppress or adjust confidence, but .kodiai.yml does not yet expose explicit repo doctrine such as API compatibility rules, migration requirements, feature-flag requirements, or raw-SQL boundaries.

---

Milestone 1: Extract the current implicit plan

Goal: make every review run explain what Kodiai intended to do before it does it.

S01: Add ReviewPlan and render it in Review Details risk:high depends:[]

After this: each review has a typed plan showing tier, scope, context sources, enabled gates, budgets, and publish policy.

Implementation targets:

• Add src/review-orchestration/review-plan.ts or similar.
• Build the plan in src/handlers/review.ts after config, diff analysis, routing, timeout-risk, large-PR triage, and structural-impact decisions are known.
• Include:
  • taskType: review.full / review.small-diff
  • routingReason: current tiny-diff / standard
  • changed-file scope: full, abbreviated, mention-only
  • timeout/max-turn budget from resolveReviewMaxTurnsOverride() and timeout estimator
  • context sources: retrieval, review precedents, wiki, active rules, structural impact, graph blast radius, canonical code
  • gates enabled: enforcement, feedback suppression, claim classifier, external-claim filter, guardrails, graph validation, prioritization
  • publish tools from the existing tool availability contract
• Add a compact Review Details section via formatReviewDetailsSummary().
• Record the plan hash in telemetry and knowledge configSnapshot.

S02: Put graphValidation in config schema risk:medium depends:[S01]

After this: the existing graph-amplified verification path can actually be enabled and audited.

Implementation targets:

• Add review.graphValidation.enabled to src/execution/config.ts.
• Document it in docs/configuration.md with the caveat that it only runs when graphBlastRadius exists.
• Add a config test proving .kodiai.yml preserves the field.
• Add a Review Details line showing whether graph validation was enabled, unavailable, skipped, or applied.

S03: Extract the reducer from review.ts risk:high depends:[S01]

After this: post-processing produces a structured reducer result instead of mutating processedFindings through many local steps.

Implementation targets:

• Create src/review-orchestration/reducer.ts around the current sequence:
  • applyEnforcement()
  • evaluateFeedbackSuppressions()
  • classifyClaims()
  • demoteExternalClaimSeverities()
  • config/incremental/large-PR suppression
  • applyClusterScoringWithDegradation()
  • filterExternalClaims()
  • runGuardrailPipeline()
  • graph validation
  • prioritizeFindings()
• Return kept, suppressed, rewritten, deprioritized, lowConfidence, and auditEvents.
• Keep current behavior first; the point is observability and testability before changing policy.

---

Milestone 2: Move from published comments to candidate findings

Goal: let Kodiai verify hypotheses before they are visible on GitHub.

S04: Add a candidate-finding MCP tool risk:high depends:[S03]

After this: the agent can submit draft findings to Kodiai without publishing inline comments immediately.

Implementation targets:

• Add an MCP server similar to src/execution/mcp/inline-review-server.ts, but for record_candidate_finding.
• Candidate schema:
  • filePath, startLine, endLine
  • claim
  • evidence[]
  • impact
  • suggestedFix
  • severity, category, confidence
  • sourceLane
• Store candidates in the workspace result or a sidecar artifact read by executor.ts, not in GitHub comments.
• Keep create_inline_comment available during migration behind a plan flag.

S05: Publish only reducer-approved candidates risk:high depends:[S04]

After this: GitHub comments are emitted after claim verification, dedupe, confidence filtering, and prioritization.

Implementation targets:

• Move inline publication later in review.ts.
• Use the reducer result to call the existing GitHub review/inline publication path.
• Keep Review Details counts for candidates generated, verified, suppressed, rewritten, and published.
• Update knowledge writes to store reducer-approved and suppressed candidates with reasons.

S06: Extend guardrails from title-only to evidence bundles risk:medium depends:[S04]

After this: guardrails classify the actual claim and evidence, not just the first-line title extracted from an inline comment.

Implementation targets:

• Extend ReviewFinding in src/lib/guardrail/adapters/review-adapter.ts or add a new candidate adapter.
• Let extractClaims() inspect claim, impact, and evidence summaries.
• Keep the current title adapter as the legacy path until S05 is stable.

---

Milestone 3: Add specialist lanes using existing task routing

Goal: specialize review work without adding uncontrolled parallelism.

S07: Add lane definitions as plan entries risk:medium depends:[S01,S04]

After this: lanes are selected by ReviewPlan, not by ad hoc prompt branches.

Implementation targets:

• Add task types in src/llm/task-types.ts, for example:
  • review.lane.correctness
  • review.lane.security
  • review.lane.docs-config
  • review.lane.dependency
• Route them through src/llm/task-router.ts and .kodiai.yml models overrides.
• Each lane gets its own budget and context request, but all lanes output the same candidate-finding schema.

S08: Start with docs/config truthfulness lane risk:medium depends:[S07]

After this: Kodiai has one low-blast-radius specialist lane proving the lane/reducer model.

Why this lane first:

• The repo already has truthfulness drift prompt sections.
• The blast radius is low compared with correctness/security.
• It directly covers Kodiai's own recurring failure mode: stale verifier names, config keys, prompts, runbooks, and Review Details text.

Implementation targets:

• Trigger on docs, config, scripts, verifier, deployment, and runbook paths.
• Feed .kodiai.yml, docs/configuration.md, relevant script names, and changed file diffs.
• Output only candidate findings; publish only after reducer approval.

S09: Add correctness/security lanes behind risk triggers risk:high depends:[S08]

After this: high-risk PRs can receive focused checks without every PR paying the cost.

Implementation targets:

• Security lane triggers from existing risk signals: auth/token/secret paths, dependency manifests, infra/deploy files, SQL/migrations, unsafe code patterns.
• Correctness lane triggers from source changes, structural-impact fanout, retries, cache, error handling, migrations, and high timeout/risk scores.
• Reuse structuralImpactForReview, graphBlastRadius, canonical code evidence, and review precedents as lane context.

---

Milestone 4: Make verification stronger than generation

Goal: explicitly try to disprove candidate findings before publishing.

S10: Add candidate verification status risk:high depends:[S05]

After this: each candidate is verified, partially_verified, unverified, or disproven before publication.

Implementation targets:

• Add a verifier step after candidates are collected and before reducer publication.
• Deterministic checks first:
  • line/path still exists in changed diff
  • claim references code visible in diff, PR body, commits, structural impact, canonical code, or retrieved context
  • external-only claims are demoted or suppressed using the existing classifier/filter logic
• Optional LLM fallback uses TASK_TYPES.GUARDRAIL_CLASSIFICATION or a new non-agentic verifier task.
• Default publication policy: suppress disproven, suppress low-confidence unverified, allow partially_verified only for high-severity risk with clear uncertainty.

S11: Add counterfactual templates to verifier, not final comments risk:medium depends:[S10]

After this: Kodiai checks retries, stale cache, partial failure, empty data, concurrency, migration/runtime compatibility, and API compatibility where relevant.

Implementation targets:

• Select scenario templates from diff/risk signals and file categories.
• Store scenario IDs in candidate evidence metadata.
• Do not dump scenario prose into GitHub comments unless it directly supports the finding.

S12: Dedupe and disagreement detection in reducer risk:medium depends:[S10]

After this: multiple lanes can disagree without producing duplicate or overconfident comments.

Implementation targets:

• Dedupe by file/line/claim fingerprint plus semantic title fingerprint.
• Merge evidence from multiple lanes into one candidate.
• If lanes disagree on severity or validity, lower confidence and require verifier confirmation before publishing.
• Render disagreement counts in Review Details only.

---

Milestone 5: Make repo memory explicit

Goal: convert learned behavior into review contracts instead of only prompt hints.

S13: Expand .kodiai.yml review doctrine risk:medium depends:[S01]

After this: repositories can declare invariants Kodiai should enforce.

Implementation targets:

• Extend review or add reviewContracts with fields such as:
  • apiCompatibility
  • migrationRequirements
  • performanceBudgets
  • forbiddenPatterns
  • requiredTracing
  • featureFlagRequirements
  • docsUpdateRequiredFor
• Feed these through formatActiveRulesSection() or a parallel static-rules section.
• Include active contract IDs in ReviewPlan and Review Details.

S14: Tie generated rules to reducer decisions risk:medium depends:[S03,S13]

After this: generated rules are not just prompt text; they are auditable signals in kept/dropped decisions.

Implementation targets:

• Attach active rule IDs to candidates that cite them.
• Store rule IDs with finding records or reducer audit records.
• Use feedback/suggestion cluster models to retire noisy generated rules.

---

Milestone 6: Roll out with shadow mode and hard metrics

Goal: prove better reviews before increasing comment volume or cost.

S15: Add shadow-mode lanes risk:high depends:[S07,S10]

After this: Kodiai can run lanes and verification without publishing lane findings.

Implementation targets:

• ReviewPlan includes lane.publishMode: shadow | publish.
• Store candidate and reducer stats without GitHub comments.
• Compare baseline published comments with shadow candidates.

S16: Add orchestration telemetry risk:medium depends:[S15]

After this: the project can answer whether orchestration improved quality.

Implementation targets:

• Add telemetry for:
  • plan tier
  • lane durations/costs
  • candidates generated
  • candidates verified/disproven/suppressed/published
  • reducer drop reasons
  • graph/canonical-code evidence usage
  • comment acceptance/reaction history
• Reuse src/telemetry/types.ts patterns and existing prompt-section/cost telemetry.

S17: Add review tiers only after shadow evidence risk:medium depends:[S16]

After this: Fast/Standard/Deep/Critical modes are backed by measured cost and signal, not just labels.

Suggested mapping:

• Fast: current review.small-diff plus existing reducer.
• Standard: current review.full plus reducer and selected truthfulness lane.
• Deep: correctness/security lanes, structural-impact/canonical-code emphasis, verifier pass.
• Critical: manually requested, stricter confidence gate, more lanes, no auto-blocking until proven.

---

Recommended first PR sequence

1. Add ReviewPlan and Review Details rendering.
2. Add review.graphValidation to config schema and docs.
3. Extract the current post-processing gates from review.ts into a reducer module with no behavior change.
4. Add candidate-finding output behind a feature flag or shadow mode.
5. Add the docs/config truthfulness lane first.
6. Add verifier status and reducer publication policy.
7. Only then add correctness/security lanes and review tiers.

This makes the issue less about copying Claude's multi-agent shape and more about Kodiai's actual next step: formalize the orchestration and verification system that already exists, then let specialist lanes feed it.