Measure NIST's three probes on the corpus: they score it well

[Silentpartnercoding]

The discriminating experiment from #107 is no longer outstanding. NIST's probe code unmodified, judge gpt-4.1, 11 citation instances per probe, 33 verdicts, zero parse errors.

Probe	Mean score
Citation faithfulness	0.82
Citation completeness	0.95
Citation sufficiency	0.86

Root count over the same three citations: 1. That is the outcome named in advance as supporting a fourth dimension.

A claim I got wrong, corrected rather than quietly dropped

#107 said the probes are structurally unable to see the derivation. They are not.

8 of the 33 rationales explicitly discuss it, in the judges' own words:

"The source fully supports the assertion that the trade press article is a secondary source that derives its data from the original Meridian note and did not conduct independent measurements."

The judges read the derivation. The corpus states it and they report it back.

And not one of the nine deductions is because of it. Every verdict below full marks is a scope mismatch inside a single citation — a detail absent from that source, or a sentence mentioning "review literature" when the cited source is the trade press. None lowers a score because three citations share a root, because each probe scores one citation against one source and none emits a count of independent sources.

The accurate statement is narrower and less comfortable:

A judge can notice derivation and still have nowhere to put it.

That's a harder problem than blindness — a fourth judge doesn't fix it unless the dimension exists in the output.

Weaknesses, recorded with the result

• One judge, one run, no variance estimate.
• The section was not written by NIST's generator. It sends reasoning_effort alongside temperature; OpenAI's reasoning models reject a non-default temperature and its chat models reject reasoning_effort, so the generator can't run against the public API without editing NIST's code. The probes are unaffected and untouched — call_judge sends neither. Section written by gpt-4.1 from the same three documents.
• The section is unusually careful, stating the derivation outright. Weak test in one direction, strong in another: even spelled out, no probe converts it into a finding.
• First attempt returned 11/11 PARSE_ERROR on completeness — HTTP 429, not disagreement. The dispatcher fires judge calls concurrently and blew a 30k TPM limit. Recorded because a rate-limited mean of 0.0 looks exactly like a disputed one; check num_parse_error before reading any mean.

Also

independence_demo.py no longer prints that no citation is unfaithful, incomplete or overreaching — the measurement contradicts it. Some are, mildly, for unrelated reasons.

🤖 Generated with Claude Code