feat: native Inspect AI hook for PRML pre-registration (v0.2.0) - #19
Merged
Conversation
Adds FalsifyHooks, registered via the inspect_ai setuptools entry point, so it is discovered automatically once installed (same mechanism as Inspect's MLflow and W&B example hooks). Set FALSIFY_PRML to a committed .prml.yaml and run inspect eval as usual: at each task end the hook reads the realised metric, checks it against the committed threshold, confirms the run's identity (model, dataset, task) matches the pre-registration, and writes a .prml-receipt.json with a PASS / FAIL / TAMPERED verdict. Observe-only by default; FALSIFY_PRML_STRICT=1 fails the run on a non-PASS verdict (a CI gate). New in core: verify_live, verify_observation, load_committed_manifest. The file-based verify_eval_log API and the CLI are unchanged; inspect_ai stays an optional dependency. CI now installs the inspect extra so the hook tests run. Also corrects a stale EU AI Act date in the README (high-risk to 2 Dec 2027). Verified end-to-end against a real mockllm/model eval (entry-point discovery, PASS, and TAMPERED paths) plus 30 passing unit tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native Inspect AI hook for PRML pre-registration
Turns
falsify-inspectinto a proper Inspect extension. Instead of calling themanual API, you point
FALSIFY_PRMLat a committed.prml.yamland runinspect evalas usual; the hook does the rest.What it does
inspect_aisetuptools entry point, so Inspect discoversit automatically once installed (same mechanism as Inspect's MLflow / W&B
example hooks).
against the committed threshold, confirms the run's identity (model, dataset,
task) matches the pre-registration, and writes a
*.prml-receipt.jsonwith aPASS/FAIL/TAMPEREDverdict.TAMPERED= the run did not match the pre-registration (e.g. a swappedmodel), so the claim cannot be quietly changed after seeing results.
FALSIFY_PRML_STRICT=1fails the run on a non-PASSverdict (a CI gate).
Scope and safety
verify_eval_logAPI and thefalsify-inspectCLI areunchanged.
inspect_aistays an optional dependency; the base package stillimports without it.
verify_live,verify_observation,load_committed_manifest.inspectextra so the hook tests actually run.Verification
mockllm/modeleval: entry-point discovery, a PASSrun, and a TAMPERED run all confirmed.
Version bumped to 0.2.0. Not yet published to PyPI.
🤖 Generated with Claude Code