Seven principles for writing rubrics that an LLM grader cannot pass by default.
日本語版: README.ja.md
If you leave a grader alone, it passes everything.
This is the default failure, not the exception.
Write "check that the article uses images that fit the context" and the grader skims the article, writes one line saying the images fit, and stops. It can write that without opening a single source. The first draft passes. The revision loop never runs once.
Write "produce a table mapping each image filename to the heading or paragraph it illustrates; any image with no match fails" and the grader has to produce evidence.
Only the second grader finds the generic stock photo.
A rubric is not a rule book. It is the ruler the grader measures with. These seven principles are about making that ruler hard to fake.
Each criterion must be decidable by counting, subtracting, cross-referencing, or citing a path and line number. "Appropriate", "clear", "compliant" cannot be decided — they can only be asserted.
Failure mode: the grader restates the criterion as a verdict. "The sources are appropriate." Nothing was checked.
Require concrete evidence before a pass is issued: the page actually fetched, the count actually counted, the file:line reference, the output of the command that was run.
A grader that is too strict costs you one extra loop. A grader that is too lenient ends the loop with the bad version still in place. The costs are not symmetric.
Real example.
tirth8205/code-review-graphissue #314 reports that the tool's injected instructions make models faster but measurably worse at understanding code. On 2026-07-27 the maintainer confirmed it still reproduces and noted: "The existing 189 instruction-generation tests pass, but they do not assert these safeguards." 189 passing tests, and not one of them tested the property that mattered. Passing checks are not evidence unless they check the thing.
Naming a specific command means that when the command is unavailable, nothing happens and the check silently passes.
State what counts as proof. Let the grader choose the means.
Failure mode:
run npm testin a repo with no npm. The step is skipped, the log is clean, the criterion is reported as met.
Every criterion has a cheap way to appear satisfied. Name it and forbid it explicitly.
Without a line like "a syndicated repost, a roundup article, or a search-result snippet does not count as verification", a dead primary source quietly becomes a secondary source and the work passes.
A useful habit: for each criterion, ask "what is the laziest thing that would make this look done?" and write that down as a disqualifier.
The grader's explanation is the writer's only signal. An unstructured paragraph produces an unfocused revision.
Force the shape:
Verdict: FAIL (2 of 7 criteria)
- [Criterion 3] 2 independent clusters found, 3 required. Three sources trace to the
same press release → add one source from a different originator.
- [Criterion 5] 2 of the quoted passages were not matched against the original text →
re-fetch the URLs, or drop the quotation marks and label it a paraphrase.
Pass verdicts need one line too, stating what was inspected. "Looks fine" is not a pass.
Without this, the grader attacks cosmetic formatting, pre-existing unrelated problems, and things outside the requested scope. The revision loop fills with noise and stops being a signal.
List what is out of bounds, and make the grader self-check before raising something: "which criterion does this fall under? If none, stay quiet."
Any deliverable that states a number or a firm conclusion must state, in the same document, the conditions under which that number does not hold. If those conditions cannot be written down, the number should not be published.
Without this, the grader checks that the number has a source and passes it — never asking what the number fails to measure. The moment a limitation moves to an appendix, a verbal caveat, or "we'll add it later", the limitation does not exist.
A good example and a residual gap, in the same repository.
code-review-graph's README does this well. It states plainly that its 376× best case "is not the typical result", that its 1.0 recall figure is "a circular upper bound, not 100% recall" because the ground truth is derived from the same graph the predictor walks, and that its honest co-change evaluation mode currently returns zero predictions and therefore "is not yet a usable measurement". It publishes its own weak numbers — search MRR 0.35, flow detection 33% recall — in a "Limitations and known weaknesses" section. On 2026-08-02 the maintainer committeddocs(benchmarks): replace unverified claims with measured numbers.The residual gap shows why principle 7 has to be enforced mechanically. The headline "~65× token reduction" is measured against reading the entire corpus — a baseline the README itself concedes "no real agent pays". It points readers to a realistic baseline benchmark and cites the result path
evaluate/results/<repo>_agent_baseline_*.csv. As of 2026-08-05 no such file exists in the repository — only the script that would produce it. The honest comparison is described, not published.(Both observations are checkable against the public repository. Neither is a claim about intent.)
If the request is "research the market", the rubric goes all the way down to "at least 3 independent source clusters from tiers 1–3, at least one of them an official primary source, every figure carrying document name, year, and table number."
A rubric written at the same level of detail as the request has given the grader nothing to do.
Do not write criteria from a blank page.
Take one past deliverable you know was good, have the model analyse why it was good, and convert that analysis into criteria. This reliably beats writing from scratch, because the criteria come from something that actually worked rather than from what sounds reasonable.
Two rubrics built on these principles, generalised from production use:
examples/research-quality.md— grading research and analysis outputexamples/strategy.md— grading strategic recommendations
(Principle 7, applied to this repository.)
- These principles were derived from grading research, strategy, and content deliverables produced by LLM agents in a small consultancy. They are not validated for code review, safety evaluation, or academic assessment.
- There is no benchmark behind them. The evidence is operational: rubrics written this way catch failures that prose criteria did not, in one organisation's workflow. That is a weaker claim than a measured one, and it is the only claim being made.
- Principles 1–6 are an adaptation of the grader-design guidance in Anthropic's claude-cookbooks (
managed_agents/CMA_verify_with_outcome_grader.ipynb). Principle 7 was added on 2026-08-05 after evaluating an external repository whose headline metric rested on a baseline it had itself already disclaimed. - Stricter rubrics cost more. Every additional evidence requirement is another revision loop. If your deliverables are low-stakes, this is overhead.
MIT © HIGHDEF INC. — https://highdef.jp