System-level lint for multi-agent harnesses. Catches the structural traps single-file linters miss — including the LLM-when-you-should-use-code patterns that burn tokens. agnix lints your files. Harness Diagnostic lints your system.
File-level linters check whether each CLAUDE.md, SKILL.md, or AGENTS.md conforms to its own schema. They cannot see the failures that live between files — the FAIL gate without a cascade actor, the 91-minute citation step that should have been 30 seconds of Python, the drafter that saves its output and then dies producing a report.
This catalog enumerates 21 such gaps observed while building paper-maker, a 15-agent academic-paper harness (v1.0 → v1.2; private companion harness, not yet public). Each gap is mapped onto a five-layer architectural model adopted from agentic-system reliability research (arXiv 2604.08906 and its 409-bug analysis).
Roughly a third of the catalog (6 of 21 gaps) is a direct cost regression — not a quality issue, a bill issue. The flagship case:
paper-maker's
pm-citation-formatteronce ran for 91 minutes on Claude Opus without producing an artifact. It had been asked to verbatim-map 60 facts against 31 KB of body text against 48 reference items in one LLM call. The transformation is deterministic — regex plus a dictionary lookup. After splitting the agent into a Python phase + a short LLM review, the same work takes 30 seconds.
That is not a 2× speedup. It is a ~99% cost reduction on one of the most expensive agents in the pipeline. The pattern — deterministic work routed through an LLM — is invisible to file-level linters because each agent definition is internally well-formed.
| Gap | Cost effect |
|---|---|
| HD-003 Deterministic work routed through an LLM ★ | The flagship cost trap. 91 min Opus tool-call loop → 30 sec Python. |
| HD-007 Parallel fanout not parameterized by content size | Prevents token burst on large jobs; avoids rate-limit retries that double-pay. |
| HD-009 No per-agent wall-clock budget | Hard limit caps runaway agents that loop on tool calls. |
| HD-011 Writer agent has no per-call content cap | A 58-minute drafter call becomes three 15-minute sub-calls — shorter total wall-clock and smaller retry unit on failure. |
| HD-018 No progress instrumentation | Without it, users kill long-running jobs and restart from zero, paying tokens twice. |
| HD-020 Uniform model assignment | Roles doing deterministic / pattern-matching work can downgrade to Sonnet — meaningful savings on long pipelines. |
A multi-agent harness with five or more roles is rarely badly designed at the file level. The way it loses money is by routing the wrong kind of work to an LLM, by never bounding how much work a single call gets, or by being so opaque that operators kill jobs and re-pay.
v0.2 sorts the catalog into three tiers so the headline contribution is not buried under standard reliability hygiene. Tier classification is independent of severity — a Tier 1 rule can be Major (HD-010), and a Tier 2 rule can be Blocker (HD-001).
| Tier | What it is | Count | Examples |
|---|---|---|---|
| Tier 1 — Harness-specific novel patterns | Failure modes that arise from multi-agent LLM orchestration with no clean classical analog. The catalog's headline contribution. | 6 | HD-003 ★ Deterministic work routed through an LLM · HD-007 Parallel fanout not size-parameterized · HD-010 Report-first convention loses artifacts · HD-011 Per-call content cap missing |
| Tier 2 — Standard reliability practices applied to agent harnesses | Well-known reliability patterns (gate routing, schema enforcement, timeout budgets, progress instrumentation) restated for the harness context. Harness builders frequently skip them because the format is new. | 12 | Cascade actor after FAIL gate · Required scope field gate · Wall-clock budget · Progress instrumentation |
| Tier 3 — Minor / observational | Configuration flexibility, model-choice cost flags, resolved patterns kept for completeness. | 3 | Uniform venue threshold · Uniform model assignment · Counter-evidence (resolved) |
Severity breakdown (parallel axis): Blocker 3 · Major 9 · Medium 7 · Minor 2.
Full catalog with primary-source quotes inline: _workspace/03_harness_gap_catalog.md.
| Layer | Concern |
|---|---|
| L1 Orchestration | phase ordering, gate handlers, cascade actors, branching |
| L2 Intelligence | model selection, LLM-vs-code boundary, reasoning depth |
| L3 Knowledge | evidence flow, retrieval gates, schema enforcement, incremental updates |
| L4 Action | tool use, external side effects, build output routing |
| L5 Infrastructure | lifecycle, runtime budgets, progress, persistence, UX surface |
In paper-maker, 10 of 21 gaps belong to L1 (heavy orchestration), and 5 belong to L5 (operational time — where most cost-saving rules live). The catalog reports its own layer skew so builders can see where the harness shape concentrates risk.
A Python prototype, standard library only:
python3 cli/diagnose.py /path/to/your/harness/root
# or for JSON output:
python3 cli/diagnose.py /path/to/your/harness/root --jsonThe CLI discovers CLAUDE.md, SKILL.md (under .claude/skills/ or root), and agent definitions (.claude/agents/*.md or agents/*.md), then applies eight of the 21 rules — the ones reliably detectable with regex plus section-presence checks: HD-001, HD-002, HD-003, HD-006, HD-009, HD-010, HD-012, HD-018. Pay particular attention to FAILs on HD-003 — that one alone has cost-class consequences.
For the thirteen rules that require reasoning over routing intent, cross-file artifact tracing, or escape-valve semantics:
- Copy
_workspace/diagnose.prompt.mdinto Claude Code, Codex CLI, Cursor, or any LLM with file-read tools. - Provide the path to your harness root.
- Read the markdown report.
Used together (static for the 8, NL for the 13), Options 1 + 2 cover all 21 rules.
If you prefer to apply the rules by hand, _workspace/03_harness_gap_catalog.md gives the detection pattern, remediation, and a primary-source quote for each gap.
Two reports run the v0.1 catalog against the seed harness — one in each direction.
_workspace/04_smoke_test_paper_maker.md(PASS-mode verification, paper-maker v1.2) — 19 PASS · 2 WARN · 1 N/A · 0 FAIL. Demonstrates that the rules recognize the remediation language in the post-fix state. On its own this is weak evidence._workspace/04b_v10_simulated_diagnostic.md(FAIL-mode demonstration, paper-maker v1.0 reconstructed) — 18 FAIL · 0 PASS · 2 WARN (persisting) · 1 N/A. The 18 rules that PASS on v1.2 produce FAIL on the reconstructed v1.0 state. The contrast is the minimum bar a static-pattern catalog must clear to be more than decoration.
Both reports are internal to the lineage that produced the catalog. The strongest available evidence — diagnostics on harnesses the rules were never extracted from — is the v0.2 gate condition.
- Not a replacement for file-level linters. Use agnix (423 rules across 9 tool prefixes as of 2026-05-14) for per-file schema validation. Harness Diagnostic operates one layer above.
- Not a generalization claim from one harness. paper-maker is orchestration-heavy. Data pipelines, deploy pipelines, and CRDT collaboration harnesses will surface different layer distributions. Inbound diagnostics on independent harnesses are how the catalog earns generalization.
- Not a list of failure causes. It is a list of structural absences — the things missing from how the harness is described.
- Not a hard cost estimator. The cost-impact column above describes direction and magnitude class, not dollar-precise predictions. Your harness, your model mix, your token volumes.
- v0.1 (2026-05-14) — initial catalog from paper-maker v1.0 → v1.2. 21 gaps, five layers, flat structure, one diagnostic prompt, one smoke test.
- v0.1.1 (2026-05-14) — epistemic-honesty pass: smoke test reframed as PASS-mode verification; companion FAIL-mode reconstruction added; citations sharpened.
- v0.2 (2026-05-14) — three-tier reorganization (Tier 1: 6 novel · Tier 2: 12 applied hygiene · Tier 3: 3 minor); primary-source quotes inline in Tier 1 and Tier 2 cards; static analyzer prototype (
cli/diagnose.py) shipped covering eight rules. No rule additions or removals. - v0.3 (planned) — first inbound diagnostic on a third-party harness; new gaps that surface get added.
- v1.0 (planned) — three or more independent harnesses contribute gaps; tier and layer classification stabilize.
This catalog grew out of two internal diagnostic notes written while iterating paper-maker (private companion harness):
_diagnostic_v1.md— 16 gaps in interface and gate definition (2026-05-13)._diagnostic_v2.md— 5 gaps in operational time, deterministic-versus-LLM work, and output-first convention (2026-05-14, after the 91-minute failure).
Those notes will be opened alongside paper-maker if and when it goes public.
External corroboration:
- Zhang, X., Zhang, H., Tan, S. H., 2026-04-10. "Dissecting Bug Triggers and Failure Modes in Modern Agentic Frameworks: An Empirical Study." arXiv:2604.08906 — 409 fixed bugs across five representative agentic frameworks; the five-layer abstraction (orchestration to infrastructure) used here.
- agnix — file-level linter (v0.26.0, 423 rules across 9 tool prefixes as of 2026-05-14); positioning anchor for what Harness Diagnostic deliberately does not cover.
- Augment Code, 2026. "Harness Engineering for AI Coding Agents: Constraints That Ship Reliable Code." augmentcode.com/guides — names the same deterministic-vs-LLM boundary that HD-003 codifies.
- Fowler, M., 2026. "Harness engineering for coding agent users." martinfowler.com/articles — frames the harness as the system layer above any single agent definition.
- OpenAI, 2026. "Harness engineering: leveraging Codex in an agent-first world." openai.com/index — Plan-Execute-Verify pattern overlaps the L1 Orchestration concerns here.
If you ran the diagnostic on your own harness and found:
- A new structural gap not in the catalog — open an issue with the gap symptom, the file evidence, and the remediation pattern you applied.
- A rule that misfired (false positive / false negative) on your harness — open an issue describing the harness shape and the failure mode.
- A rule that genuinely doesn't apply to a class of harness — propose a "scope: not applicable when …" clause.
- A cost outcome from applying a remediation (e.g., "HD-003 split saved us $X/run on Y workload") — those numbers harden the cost column.
Catalog evolution is the entire point of v0.x.
MIT. Use it, fork it, integrate it into your own tooling.