diff --git a/interop/nist-agentic-research-probes/README.md b/interop/nist-agentic-research-probes/README.md index c6b48d5..3ac87bf 100644 --- a/interop/nist-agentic-research-probes/README.md +++ b/interop/nist-agentic-research-probes/README.md @@ -73,29 +73,29 @@ Deterministic, no model, no network: The independence side of the proposed dimension needs no judge. -## What is NOT demonstrated +## The discriminating experiment — run, see `RESULTS.md` -**The three existing probes have not been run on this corpus.** They are -LM-judge probes and require an OpenAI-compatible endpoint. No such run has been -performed, so this directory does not claim that faithfulness, completeness and -sufficiency score well here. It claims only that they are structurally unable to -see the derivation, which follows from their per-citation design. +Run on 18 August 2026 with NIST's probe code unmodified, judge `gpt-4.1`: -This is the discriminating experiment and it is outstanding: - -1. Ingest `corpus/` with NIST's pipeline. -2. Ask a question that pulls all three documents into one section. -3. Run the three probes unchanged. - -**The result that supports a fourth probe:** all three score well while the root -count is 1. **The result that refutes it:** any of the three flags the -derivative citations. In that case one of the existing probes already covers -this and no fourth dimension is warranted — and that finding should be recorded -here rather than discarded. - -Until that run happens, the argument above is structural, not empirical. A -proposal to NIST on this basis would be asserting a measurement nobody made, -which is the failure this project exists to catch. +| Probe | Mean score | +|---|---| +| Citation faithfulness | 0.82 | +| Citation completeness | 0.95 | +| Citation sufficiency | 0.86 | + +Root count over the same three citations: **1**. That is the outcome that +supports a fourth dimension, and it was named in advance. + +**One claim above needed correcting.** An earlier version of this file said the +probes are *structurally unable to see* the derivation. That is too strong. 8 of +the 33 verdict rationales explicitly discuss it — the judges read the derivation +and report it back. What none of them do is act on it: every deduction is a +scope mismatch inside a single citation, because each probe scores one citation +against one source and no probe emits a count of independent sources. + +The accurate statement is narrower and less comfortable: a judge can notice +derivation and still have nowhere to put it. `RESULTS.md` carries the evidence, +the weaknesses, and what would still refute the case. ## Provenance of the claims here diff --git a/interop/nist-agentic-research-probes/RESULTS.md b/interop/nist-agentic-research-probes/RESULTS.md new file mode 100644 index 0000000..55a624a --- /dev/null +++ b/interop/nist-agentic-research-probes/RESULTS.md @@ -0,0 +1,85 @@ +# Measurement: what NIST's three probes score on this corpus + +Run 18 August 2026. Judge model `gpt-4.1`. NIST's probe code unmodified. +Raw verdicts in `probe-results.json`; runner in `run_probes.py`. + +## Result + +| Probe | Mean score | Verdicts below full marks | +|---|---|---| +| Citation faithfulness | **0.82** | 2 partially supported, 1 not supported | +| Citation completeness | **0.95** | 2 minor omission | +| Citation sufficiency | **0.86** | 3 minor overreach, 1 significant overreach | + +Eleven citation instances per probe, 33 verdicts, **zero parse errors**. + +Meanwhile the root count over the same three citations is **1** +(`independence_demo.py`). + +The discriminating experiment predicted in the previous README has been run and +the answer is the one that supports a fourth dimension: the three probes score +the section well while the evidence-root analysis shows a single root. + +## The sharper finding, which is not the one predicted + +The prediction was that the probes are *structurally unable to see* the +derivation. That turns out to be too strong, and the real result is more +interesting. + +**8 of the 33 rationales explicitly discuss derivation.** The judges write +things like: + +> "The source fully supports the assertion that the trade press article is a +> secondary source that derives its data from the original Meridian note and did +> not conduct independent measurements." + +So the judges *read* the derivation. The corpus states it plainly and they +report it back. + +**And not one of the nine deductions is because of it.** Every verdict below +full marks is a scope mismatch inside a single citation — a detail absent from +that particular source, or a sentence mentioning "review literature" when the +cited source is the trade press. None says the three citations are one +observation, because no probe has a verdict that could say it. + +The accurate claim is therefore narrower and worse: + +> A judge can notice derivation and still not act on it, because every probe +> scores one citation against one source. The number of independent sources is +> not a quantity any of the three produces. + +Awareness without a place to put it. That is a harder problem than blindness, +because adding a fourth judge does not fix it on its own — the dimension has to +exist in the output. + +## Honest weaknesses in this measurement + +- **One judge model, one run.** `gpt-4.1` at temperature 0. No repeats, so no + variance estimate. A different judge might score differently. +- **The report section was not written by NIST's generator.** Their report + pipeline sends `reasoning_effort` alongside `temperature` + (`exhaustive_scanner.py:104`, `pipeline.py:115`); OpenAI's reasoning models + reject a non-default temperature and its chat models reject + `reasoning_effort`, so the generator cannot run against the public OpenAI API + without editing NIST's code. The generator was written for their internal + `gpt-oss-120b` endpoint. **The probes are unaffected and unmodified** — + `probes/_judge.py::call_judge` sends only model, messages, temperature and + `response_format`. The section was written by `gpt-4.1` from the same three + documents. +- **The section is unusually careful.** It states outright that the secondary + sources derive from the original. A sloppier report would give the probes more + to catch. This makes the corpus a *weak* test in one direction and a strong + one in another: even when the derivation is spelled out in the text, no probe + converts it into a finding. +- **Probes are rate-limit sensitive.** The first attempt returned 11 of 11 + `PARSE_ERROR` on completeness. That was not the judge failing but HTTP 429 — + the dispatcher fires all judge calls concurrently and exceeded a 30k + tokens-per-minute account limit. Recorded because a mean score of 0.0 from + rate limiting looks exactly like a mean score of 0.0 from disagreement, and + anyone reproducing this should check `num_parse_error` before reading a mean. + +## What would still refute the fourth-probe case + +A probe or verdict category in the current three that lowers a score *because* +sources share a root. Nothing in these 33 verdicts does, but the search was over +one corpus and one judge. diff --git a/interop/nist-agentic-research-probes/independence_demo.py b/interop/nist-agentic-research-probes/independence_demo.py index 9d8a603..95f0435 100644 --- a/interop/nist-agentic-research-probes/independence_demo.py +++ b/interop/nist-agentic-research-probes/independence_demo.py @@ -80,8 +80,10 @@ def main() -> int: assert len(result.support_true) == 1, result.support_true assert len(CITATIONS) == 3 - print("\nthree citations, one root. No citation is unfaithful, incomplete or") - print("overreaching; the count is simply not three.") + print("\nthree citations, one root.") + print("Measured 18 Aug 2026: NIST's probes score this corpus 0.82 / 0.95 /") + print("0.86, and not one of their deductions is about the shared root.") + print("See RESULTS.md -- including where they DO remark on the derivation.") return 0 diff --git a/interop/nist-agentic-research-probes/probe-results.json b/interop/nist-agentic-research-probes/probe-results.json new file mode 100644 index 0000000..555a22d --- /dev/null +++ b/interop/nist-agentic-research-probes/probe-results.json @@ -0,0 +1,247 @@ +{ + "model": "gpt-4.1", + "question": "What thermal drift has been reported for Kestrel-7 sensor arrays under extended hold at elevated temperature, and what is the evidence for it?", + "section": "### Reported Thermal Drift of Kestrel-7 Sensor Arrays Under Extended Elevated Temperature Hold\n\nThe only reported measurement of thermal drift for Kestrel-7 sensor arrays under extended hold at elevated temperature comes from a single-site study by Meridian Instrumentation Laboratory. In this study, one Kestrel-7 array was held at 41\u202f\u00b0C for 300 hours, resulting in a mean output drift of 0.42% of full scale, with drift observed to be monotonic for the first 180 hours and stable thereafter[^1][^2][^3]. This result has been cited in both trade press and review literature, but all secondary sources derive their data from the original Meridian note and did not conduct independent measurements[^2][^3]. The laboratory explicitly cautioned that this finding is based on a single array at a single site, with no independent calibration of the reference cell during the hold, and does not establish a general population rate or temperature dependence[^1][^2][^3]. No additional or corroborating measurements have been reported in the literature to date. Thus, while a sub-percent drift has been observed, the evidence is limited to a single, narrowly scoped experiment.\n\n[^1]: 01-root-observation.md \n[^2]: 02-derivative-trade-press.md \n[^3]: 03-derivative-review.md", + "probes": [ + { + "probe": "citation_faithfulness", + "mean_score": 0.8181818181818182, + "summary": { + "num_supported": 8, + "num_partially_supported": 2, + "num_not_supported": 1, + "num_citations_checked": 11 + }, + "verdicts": [ + { + "citation_id": 1, + "verdict": "SUPPORTED", + "score": 1.0, + "rationale": "The sentence accurately summarizes the key findings of the source: the temperature (41\u202f\u00b0C), the duration (300 hours), the mean drift (0.42% of full scale), and the drift behavior (monotonic for the first 180 hours, flat thereafter). The source states: 'One Kestrel-7 sensor array, serial K7-0442, was held at 41 \u00b0C for 300 hours... Mean drift over the 300-hour hold was 0.42 % of full scale... Drift " + }, + { + "citation_id": 2, + "verdict": "SUPPORTED", + "score": 1.0, + "rationale": "The source passage states: 'Meridian Instrumentation Laboratory has reported thermal drift of 0.42 % of full scale for a Kestrel-7 sensor array held at 41 \u00b0C for 300 hours... The laboratory recorded drift as monotonic through roughly the first 180 hours of the hold, after which output remained flat.' This directly supports the claim in the sentence regarding the experimental conditions (one array," + }, + { + "citation_id": 3, + "verdict": "PARTIALLY_SUPPORTED", + "score": 0.5, + "rationale": "The source supports the claim that a 0.42% drift over 300 hours at 41\u202f\u00b0C was reported for a single array by Meridian Instrumentation Laboratory. However, the source does not mention that the drift was monotonic for the first 180 hours and stable thereafter. This additional detail is not present in the cited source passage. Therefore, while the core measurement and context are supported, the specif" + }, + { + "citation_id": 2, + "verdict": "NOT_SUPPORTED", + "score": 0.0, + "rationale": "The source passage provides details about the original measurement and notes that Sensor Industry Weekly did not conduct its own measurement, but it does not state or imply that the result has been cited in other trade press or review literature, nor does it discuss the provenance of secondary sources. The claim in the marked sentence goes beyond what is stated in the source passage by asserting t" + }, + { + "citation_id": 3, + "verdict": "SUPPORTED", + "score": 1.0, + "rationale": "The source passage confirms that the 0.42% drift figure comes from the Meridian Instrumentation Laboratory and that it has been 'subsequently summarised in the trade press.' The passage also clarifies that Section 4.3 is a literature summary and that no new measurements were performed for the review, with all content derived from the original Meridian note. This supports the claim that secondary s" + }, + { + "citation_id": 1, + "verdict": "SUPPORTED", + "score": 1.0, + "rationale": "The claim in the marked sentence is that the laboratory explicitly cautioned about the limitations of the measurement: that it was based on a single array at a single site, with no independent calibration of the reference cell, and that it does not establish a general population rate or temperature dependence. The source passage directly states all these points: it notes the single array, single s" + }, + { + "citation_id": 2, + "verdict": "PARTIALLY_SUPPORTED", + "score": 0.5, + "rationale": "The source passage states: \"Meridian notes that the figure comes from a single array at a single site and that the chamber reference cell was not independently calibrated during the hold. The laboratory does not present the result as a population rate.\" This directly supports the claim that the finding is based on a single array at a single site, with no independent calibration of the reference ce" + }, + { + "citation_id": 3, + "verdict": "SUPPORTED", + "score": 1.0, + "rationale": "The source passage directly supports the claim that the laboratory cautioned about the limitations of the finding: it was based on one array, one site, and a reference cell not independently calibrated. The source does not explicitly mention that the result 'does not establish a general population rate or temperature dependence,' but the context ('Readers should note the reporting laboratory's own" + }, + { + "citation_id": 1, + "verdict": "SUPPORTED", + "score": 1.0, + "rationale": "The source passage (01-root-observation.md) directly supports all the factual claims made in the preceding paragraph, which the marked citation is attached to. Specifically, it reports the single-site, single-array measurement at 41\u00b0C for 300 hours, the mean drift of 0.42% of full scale, the monotonic then flat drift pattern, and the explicit cautions about the limitations of the result (single ar" + }, + { + "citation_id": 2, + "verdict": "SUPPORTED", + "score": 1.0, + "rationale": "The source passage (02-derivative-trade-press.md) directly supports the claims attributed to it in the preceding sentences: it reports the 0.42% drift at 41\u00b0C for 300 hours, describes the monotonic then stable drift, notes the single-site/single-array limitation, the lack of independent calibration, and that the result is not a population rate. The passage also clarifies that Sensor Industry Weekl" + }, + { + "citation_id": 3, + "verdict": "SUPPORTED", + "score": 1.0, + "rationale": "The source passage directly supports the claims made in the paragraph: it reports the 0.42% drift over 300 hours at 41\u00b0C from Meridian Instrumentation Laboratory, notes that this is the most frequently cited figure, and highlights the limitations (single array, single site, no independent calibration). It also clarifies that the review itself did not perform measurements and is summarizing literat" + } + ] + }, + { + "probe": "citation_completeness", + "mean_score": 0.9454545454545454, + "summary": { + "num_complete": 9, + "num_minor_omission": 2, + "num_significant_omission": 0, + "num_misrepresentation": 0, + "num_not_found": 0, + "num_parse_error": 0, + "num_citations_checked": 11 + }, + "verdicts": [ + { + "citation_id": 1, + "verdict": "COMPLETE", + "score": 1.0, + "rationale": "The citing text and its adjacent context faithfully represent the source's findings, limitations, and epistemic boundaries. The core result and its narrow scope are both clearly conveyed, and the reader is not misled about the generalizability or certainty of the finding. No material nuance or caveat from the source is omitted in the local context." + }, + { + "citation_id": 2, + "verdict": "COMPLETE", + "score": 1.0, + "rationale": "The citing text and its immediate context accurately and completely represent the source's findings, limitations, and epistemic boundaries. There is no flattening of uncertainty, erasure of scope, or omission of important caveats. The reader is not misled about the strength, generalizability, or provenance of the result." + }, + { + "citation_id": 3, + "verdict": "COMPLETE", + "score": 1.0, + "rationale": "The citing text accurately represents the source's main findings, limitations, and epistemic boundaries, and the surrounding context addresses all key caveats and scope conditions. The only minor difference is the mention of the drift being monotonic for 180 hours and stable thereafter, which is not in the provided source passage, but this does not materially change the reader's understanding. No " + }, + { + "citation_id": 2, + "verdict": "COMPLETE", + "score": 1.0, + "rationale": "The marked sentence and its surrounding context accurately represent the source's message regarding the provenance of the data, the lack of independent measurements by secondary sources, and the limitations of the original finding. All key epistemic boundaries and limitations are present in the local context. There is no material omission or distortion." + }, + { + "citation_id": 3, + "verdict": "COMPLETE", + "score": 1.0, + "rationale": "The citing text and its context accurately and completely represent the source's message, including the provenance of the data, the lack of independent measurements, and the explicit limitations and scope. There is no material omission or distortion. The epistemic weight and boundaries are preserved, and the reader is not misled about the strength or generalizability of the finding." + }, + { + "citation_id": 1, + "verdict": "MINOR_OMISSION", + "score": 0.7, + "rationale": "The citing text and its context capture all the major limitations and epistemic boundaries stated in the source, including the single-unit, single-site, lack of independent calibration, and lack of generalizability. The only minor omissions are the explicit mention that a single unit does not characterize a production lot and that no failure threshold is established. These are secondary to the mai" + }, + { + "citation_id": 2, + "verdict": "MINOR_OMISSION", + "score": 0.7, + "rationale": "The citing text and its context faithfully represent the source's findings, limitations, and epistemic caution. The only very minor stretch is the explicit mention of 'temperature dependence,' which is not directly stated in the source but is implied by the fact that only one temperature was tested. This does not materially alter the reader's understanding, as the main limitations (single array, s" + }, + { + "citation_id": 3, + "verdict": "COMPLETE", + "score": 1.0, + "rationale": "The citing text and its context accurately and completely represent the source's message, including all key limitations, epistemic hedges, and scope boundaries. There is no material omission or distortion. The reader is not misled about the strength, generalizability, or limitations of the finding." + }, + { + "citation_id": 1, + "verdict": "COMPLETE", + "score": 1.0, + "rationale": "The citing text and its immediate context faithfully represent the source's findings, limitations, and epistemic boundaries. There is no flattening of uncertainty, erasure of scope, or omission of key caveats. The reader is clearly informed of the narrowness and limitations of the evidence. No material nuance or condition is missing." + }, + { + "citation_id": 2, + "verdict": "COMPLETE", + "score": 1.0, + "rationale": "All material details, limitations, and epistemic boundaries from the source are preserved in the citing text and its context. There is no flattening of uncertainty, erasure of scope, or omission of important caveats. The representation is faithful and complete." + }, + { + "citation_id": 3, + "verdict": "COMPLETE", + "score": 1.0, + "rationale": "The citing text, together with its immediate context, faithfully and completely represents the source's message, including all key limitations, epistemic hedges, and scope boundaries. There is no material omission or distortion. The only minor addition is the mention of monotonic drift for 180 hours and stability thereafter, which is not present in the source but does not misrepresent or overstate" + } + ] + }, + { + "probe": "citation_sufficiency", + "mean_score": 0.8545454545454546, + "summary": { + "num_fully_sufficient": 7, + "num_minor_overreach": 3, + "num_significant_overreach": 1, + "num_unsupported_fig_leaf": 0, + "num_not_found": 0, + "num_parse_error": 0, + "num_citations_checked": 11 + }, + "verdicts": [ + { + "citation_id": 1, + "verdict": "FULLY_SUFFICIENT", + "score": 1.0, + "rationale": "The claim in the citing sentence is carefully limited to the scope of the source: it describes the results of a single study, with no attempt to generalize or extrapolate. The language matches the evidence, and all sub-assertions are directly supported by the source. There is no inflation of magnitude, no causal escalation, and no orphaned sub-assertions. The sentence does not overreach the eviden" + }, + { + "citation_id": 2, + "verdict": "FULLY_SUFFICIENT", + "score": 1.0, + "rationale": "There is no evidentiary gap. The claim is carefully limited to the scope of the source: it describes the result as coming from 'this study' and does not generalize. All sub-assertions (duration, temperature, drift value, drift behavior) are directly supported by the source. There is no overreach, inflation, or extrapolation beyond what the source provides." + }, + { + "citation_id": 3, + "verdict": "SIGNIFICANT_OVERREACH", + "score": 0.3, + "rationale": "The claim's first two sub-assertions (experimental setup and drift magnitude) are directly supported by the source. However, the third sub-assertion\u2014that drift was monotonic for the first 180 hours and stable thereafter\u2014is not present in the source passage. The source only reports the total drift over 300 hours, not the time-course behavior. Thus, the citation does not fully support the claim rega" + }, + { + "citation_id": 2, + "verdict": "MINOR_OVERREACH", + "score": 0.7, + "rationale": "The source fully supports the assertion that the trade press article (Sensor Industry Weekly) is a secondary source that derives its data from the original Meridian note and did not conduct independent measurements. However, the claim also refers to 'review literature,' and the source passage does not mention review literature or provide evidence about the practices of review articles. Thus, the s" + }, + { + "citation_id": 3, + "verdict": "MINOR_OVERREACH", + "score": 0.7, + "rationale": "The source directly supports the claim that the result has been cited in review literature (this review) and in the trade press, and that these secondary sources derive their data from the original Meridian note. The source also explicitly states that it performed no new measurements. The only potential gap is that the claim refers to 'all secondary sources,' which is a broad assertion. The source" + }, + { + "citation_id": 1, + "verdict": "FULLY_SUFFICIENT", + "score": 1.0, + "rationale": "The claim in the citing sentence is a faithful and accurate summary of the limitations explicitly stated in the source. Each sub-assertion in the claim is directly supported by the source's own language. There is no extrapolation, rhetorical inflation, or unsupported sub-assertion. The claim does not go beyond the evidence; rather, it mirrors the source's caution." + }, + { + "citation_id": 2, + "verdict": "FULLY_SUFFICIENT", + "score": 1.0, + "rationale": "There is no evidentiary gap. The claim in the citing sentence is a faithful and accurate restatement of the explicit limitations and caveats provided in the source. The claim does not overreach, extrapolate, or inflate the evidence. Each sub-assertion is directly supported by the source's own language and framing." + }, + { + "citation_id": 3, + "verdict": "FULLY_SUFFICIENT", + "score": 1.0, + "rationale": "There is no evidentiary gap. The claim is a careful restatement of the limitations explicitly described in the source. The source directly supports all sub-assertions: the single array, single site, lack of independent calibration, and the lack of generalizability. There is no overreach, inflation, or extrapolation beyond the evidence provided by the source." + }, + { + "citation_id": 1, + "verdict": "FULLY_SUFFICIENT", + "score": 1.0, + "rationale": "There is no evidentiary gap. The claim in the sentence is a direct, faithful, and complete restatement of the source's findings, with no extrapolation, inflation, or causal escalation. The sentence does not generalize beyond the evidence, and all sub-assertions (the experimental setup, the result, and the observed drift pattern) are directly supported by the source." + }, + { + "citation_id": 2, + "verdict": "FULLY_SUFFICIENT", + "score": 1.0, + "rationale": "There is no evidentiary gap. The claim in the sentence is strictly limited to the scope of the source: it does not generalize, exaggerate, or escalate the findings. All sub-assertions are directly supported by the source. The sentence does not attempt to draw broader conclusions or causal inferences beyond what the source provides." + }, + { + "citation_id": 3, + "verdict": "MINOR_OVERREACH", + "score": 0.7, + "rationale": "There is no significant evidentiary gap. The claim in the citing sentence is carefully limited to the scope of the evidence provided by the source: it describes the result as coming from a single-site study, with a single array, and gives the specific drift figure and conditions. The sentence does not generalize or overstate the evidence. The only minor potential gap is the mention of the drift be" + } + ] + } + ] +} diff --git a/interop/nist-agentic-research-probes/run_probes.py b/interop/nist-agentic-research-probes/run_probes.py new file mode 100644 index 0000000..aec05df --- /dev/null +++ b/interop/nist-agentic-research-probes/run_probes.py @@ -0,0 +1,146 @@ +"""Run NIST's three probes, unchanged, over a report section citing three +documents that descend from one observation. + +Why not the whole pipeline: NIST's report generator sends `reasoning_effort` +(exhaustive_scanner.py:104, pipeline.py:115) alongside `temperature`. OpenAI's +reasoning models reject a non-default temperature and its chat models reject +reasoning_effort, so the generator cannot run against the public API without +editing NIST's code. The probes are different: `probes/_judge.py::call_judge` +sends only model, messages, temperature and response_format. They run unchanged. + +So the section is written by gpt-4.1 from the three documents, and then NIST's +faithfulness, completeness and sufficiency probes grade it. What is under test +is the probes, and they are untouched. + +To reproduce: + + git clone https://github.com/usnistgov/agentic-research-evaluation-probes + cd agentic-research-evaluation-probes + pip install -r requirements.txt + cp -r /path/to/this/directory/corpus ./crossing-corpus + cp /path/to/this/directory/run_probes.py . + OPENAI_API_KEY=... CROSSING_CORPUS=./crossing-corpus python3 run_probes.py + +Check `num_parse_error` in the output before reading any mean score. The probes +issue their judge calls concurrently and a token-per-minute limit turns verdicts +into PARSE_ERROR scored 0.0, which is indistinguishable from disagreement in the +mean alone. +""" + +import asyncio +import json +import os +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent / "src")) + +from dotenv import load_dotenv +from openai import AsyncOpenAI + +load_dotenv(Path.home() / ".config" / "odr.env") + +MODEL = os.environ.get("ODR_JUDGE_MODEL", "gpt-4.1") +# Defaults to the corpus beside this file. Override with CROSSING_CORPUS when the +# script is copied into a NIST checkout, as the run instructions describe. +CORPUS = Path(os.environ.get("CROSSING_CORPUS", + Path(__file__).resolve().parent / "corpus")) +QUESTION = ("What thermal drift has been reported for Kestrel-7 sensor arrays " + "under extended hold at elevated temperature, and what is the " + "evidence for it?") + + +async def write_section(client, docs) -> str: + listing = "\n\n".join( + f"[Source {i}] {name}\n{text}" for i, (name, text) in enumerate(docs, 1)) + prompt = ( + f"Write one section of a research report answering this question:\n\n" + f"{QUESTION}\n\n" + f"Use footnote citations in the form [^N] where N is the source number. " + f"Cite every source you rely on. Be accurate and do not overstate. " + f"Write 4-6 sentences.\n\n{listing}" + ) + r = await client.chat.completions.create( + model=MODEL, temperature=0.2, + messages=[{"role": "user", "content": prompt}]) + return r.choices[0].message.content.strip() + + +async def main() -> None: + from models import Finding, SectionResult + from citations.tracker import CitationTracker + from probes import run_probes + from research.context import ResearchContext, ResearchInfrastructure, ResearchState + from store.document_store import DocumentStore + + docs = [(p.name, p.read_text()) for p in sorted(CORPUS.glob("*.md"))] + assert len(docs) == 3, docs + client = AsyncOpenAI() + + content = await write_section(client, docs) + print("=== SECTION AS WRITTEN ===\n") + print(content) + print() + + findings = [ + Finding(citation_id=i, chunk_id=f"chunk-{i}", source_file=name, + heading="", text=text) + for i, (name, text) in enumerate(docs, 1) + ] + section = SectionResult(section_title="Reported thermal drift", + content=content, citations_used=[1, 2, 3], order=1) + + infra = ResearchInfrastructure( + document_store=DocumentStore(CORPUS), + citation_tracker=CitationTracker(), + openai_client=client, + model_name=MODEL, + ) + context = ResearchContext(infra=infra, + state=ResearchState(research_question=QUESTION)) + + # Your org's TPM limit for this model is 30k. NIST's dispatcher fires every + # probe's judge calls concurrently, which exceeds it and turns verdicts into + # PARSE_ERROR (HTTP 429). The probes are NOT modified: they are simply + # invoked one at a time with a pause, so each stays inside the window. + from probes import _PROBE_REGISTRY + for i, probe_fn in enumerate(_PROBE_REGISTRY): + if i: + print(f" ... pausing 70s to stay under the token-per-minute limit") + await asyncio.sleep(70) + print(f"running {probe_fn.__name__}") + result = await probe_fn(section, findings, context) + # The dispatcher normally stores this; we are calling the probes + # directly, so store it the same way it would. + key = getattr(result, "probe_name", None) or probe_fn.__name__ + section.probe_results[key] = result + + print("=== PROBE RESULTS (NIST code, unchanged) ===") + out = {"model": MODEL, "question": QUESTION, "section": content, "probes": []} + def field(obj, key, default=None): + if isinstance(obj, dict): + return obj.get(key, default) + return getattr(obj, key, default) + + for name, probe in section.probe_results.items(): + mean = field(probe, "mean_score") + summary = field(probe, "summary") + entry = {"probe": name, "mean_score": mean, + "summary": summary, "verdicts": []} + print(f"\n{name}: mean_score = {mean} {summary}") + for v in field(probe, "verdicts", []) or []: + vd = field(v, "verdict") + sc = field(v, "score") + cid = field(v, "citation_id") + print(f" citation [^{cid}] -> {vd} ({sc})") + entry["verdicts"].append({ + "citation_id": cid, "verdict": vd, "score": sc, + "rationale": (field(v, "rationale", "") or "")[:400]}) + out["probes"].append(entry) + + Path("probe-results.json").write_text(json.dumps(out, indent=2, default=str) + "\n") + print("\nwritten: probe-results.json") + + +if __name__ == "__main__": + asyncio.run(main())