Skip to content

feat: native Inspect AI hook for PRML pre-registration (v0.2.0) - #19

Merged
sk8ordie84 merged 1 commit into
mainfrom
feat/inspect-hook-v0.2.0
Jun 1, 2026
Merged

feat: native Inspect AI hook for PRML pre-registration (v0.2.0)#19
sk8ordie84 merged 1 commit into
mainfrom
feat/inspect-hook-v0.2.0

Conversation

@sk8ordie84

Copy link
Copy Markdown
Contributor

Native Inspect AI hook for PRML pre-registration

Turns falsify-inspect into a proper Inspect extension. Instead of calling the
manual API, you point FALSIFY_PRML at a committed .prml.yaml and run
inspect eval as usual; the hook does the rest.

What it does

  • Registered via the inspect_ai setuptools entry point, so Inspect discovers
    it automatically once installed (same mechanism as Inspect's MLflow / W&B
    example hooks).
  • At each task end: reads the realised metric from the eval log, checks it
    against the committed threshold, confirms the run's identity (model, dataset,
    task) matches the pre-registration, and writes a *.prml-receipt.json with a
    PASS / FAIL / TAMPERED verdict.
  • TAMPERED = the run did not match the pre-registration (e.g. a swapped
    model), so the claim cannot be quietly changed after seeing results.
  • Observe-only by default; FALSIFY_PRML_STRICT=1 fails the run on a non-PASS
    verdict (a CI gate).

Scope and safety

  • The file-based verify_eval_log API and the falsify-inspect CLI are
    unchanged. inspect_ai stays an optional dependency; the base package still
    imports without it.
  • New, tested core helpers: verify_live, verify_observation,
    load_committed_manifest.
  • CI now installs the inspect extra so the hook tests actually run.

Verification

  • 30 passing unit tests (PASS / FAIL / TAMPERED, strict mode, receipt).
  • End-to-end against a real mockllm/model eval: entry-point discovery, a PASS
    run, and a TAMPERED run all confirmed.

Version bumped to 0.2.0. Not yet published to PyPI.

🤖 Generated with Claude Code

Adds FalsifyHooks, registered via the inspect_ai setuptools entry point, so it
is discovered automatically once installed (same mechanism as Inspect's MLflow
and W&B example hooks). Set FALSIFY_PRML to a committed .prml.yaml and run
inspect eval as usual: at each task end the hook reads the realised metric,
checks it against the committed threshold, confirms the run's identity (model,
dataset, task) matches the pre-registration, and writes a .prml-receipt.json
with a PASS / FAIL / TAMPERED verdict. Observe-only by default;
FALSIFY_PRML_STRICT=1 fails the run on a non-PASS verdict (a CI gate).

New in core: verify_live, verify_observation, load_committed_manifest. The
file-based verify_eval_log API and the CLI are unchanged; inspect_ai stays an
optional dependency. CI now installs the inspect extra so the hook tests run.
Also corrects a stale EU AI Act date in the README (high-risk to 2 Dec 2027).

Verified end-to-end against a real mockllm/model eval (entry-point discovery,
PASS, and TAMPERED paths) plus 30 passing unit tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sk8ordie84
sk8ordie84 merged commit 0c4a885 into main Jun 1, 2026
3 checks passed
@sk8ordie84
sk8ordie84 deleted the feat/inspect-hook-v0.2.0 branch June 1, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant