feat: native Inspect AI hook for PRML pre-registration (v0.2.0)

.github/workflows/test.yml

@@ -20,7 +20,7 @@ jobs:
       - name: Install
         run: |
           python -m pip install --upgrade pip
-          pip install -e ".[dev]"
+          pip install -e ".[dev,inspect]"
       - name: Run tests
         run: pytest tests/ -v --cov=falsify_inspect --cov-report=term-missing
       - name: CLI smoke test

CHANGELOG.md

@@ -2,6 +2,30 @@
 
 All notable changes to `falsify-inspect` are documented here.
 
+## [0.2.0] — 2026-06-01
+
+### Added
+- **Native Inspect AI hook (`FalsifyHooks`).** Registered via the `inspect_ai`
+  setuptools entry point, so it is discovered automatically once installed
+  (the same mechanism as Inspect's own MLflow / W&B example hooks). Set
+  `FALSIFY_PRML=path/to/x.prml.yaml` and run `inspect eval` as usual: at each
+  task end the hook reads the realised metric, checks it against the committed
+  threshold, confirms the run's identity (model, dataset, task) matches the
+  pre-registration, and writes a `*.prml-receipt.json` with a PASS / FAIL /
+  TAMPERED verdict. Observe-only by default; set `FALSIFY_PRML_STRICT=1` to
+  fail the run on a non-PASS verdict (a CI gate).
+- `verify_live()` — verify a live `EvalLog` against a committed manifest in
+  process (no need to write the log to disk first).
+- `verify_observation()` and `load_committed_manifest()` — helpers underlying
+  the hook, exported for direct use.
+
+### Notes
+- The file-based `verify_eval_log()` API and the `falsify-inspect` CLI are
+  unchanged.
+- `inspect_ai` remains an optional dependency; the hook is only loaded when
+  running under Inspect. Verified end-to-end against a real `mockllm/model`
+  eval (entry-point discovery, PASS, and TAMPERED paths).
+
 ## [0.1.0] — 2026-05-08
 
 Initial public release.

README.md

@@ -62,6 +62,32 @@ result = verify_eval_log(
 assert result["ok"]
 ```
 
+## Quickstart — Inspect hook (automatic, recommended)
+
+As of v0.2.0 `falsify-inspect` registers a native Inspect [hook](https://inspect.aisi.org.uk/extensions.html#hooks). Pre-register a manifest, point `FALSIFY_PRML` at it, and run your eval as usual. No changes to your Inspect code:
+
+```bash
+# 1. Pre-register the claim (writes harmbench.prml.yaml)
+falsify-inspect lock \
+  --metric refusal_rate --threshold 0.95 --threshold-direction ">=" \
+  --dataset harmbench-v1 --dataset-hash sha256:abc... \
+  --model-version "claude-3.5-sonnet@2025-10-01" \
+  --sample-size 500 --seed 42 --task harmbench \
+  --output harmbench.prml.yaml
+
+# 2. Run the eval with the hook enabled
+export FALSIFY_PRML=harmbench.prml.yaml
+inspect eval harmbench.py --model anthropic/claude-3-5-sonnet-latest
+```
+
+At each task end the hook reads the realised metric, checks it against the committed threshold, confirms the run's identity (model, dataset, task) matches what you pre-registered, and writes a `harmbench.prml-receipt.json` with a `PASS` / `FAIL` / `TAMPERED` verdict. `TAMPERED` means the run did not match the pre-registration (for example a swapped model), so you cannot quietly change the claim after seeing results.
+
+The hook is observe-only by default. To make a non-`PASS` verdict fail the run (a CI gate):
+
+```bash
+export FALSIFY_PRML_STRICT=1
+```
+
 ## Quickstart — CLI
 
 ```bash
@@ -120,7 +146,7 @@ When the log was generated with a newer Inspect AI release, retry verification i
 
 Where this plugin fits in named AI governance frameworks (subcategory-by-subcategory, FULL / PARTIAL / NONE tagged):
 
-- [EU AI Act Article 12 crosswalk](https://spec.falsify.dev/eu-ai-act/article-12/) — automated-logging pattern for the 2 August 2026 deadline
+- [EU AI Act Article 12 crosswalk](https://spec.falsify.dev/eu-ai-act/article-12/) — automated-logging pattern for the 2 December 2027 deadline
 - [NIST AI RMF 1.0 crosswalk](https://spec.falsify.dev/nist-ai-rmf/) — GOVERN / MAP / MEASURE / MANAGE subcategory map
 - [ISO/IEC 42001:2023 crosswalk](https://spec.falsify.dev/iso-42001/) — AI Management System clause-by-clause evidence map
 - [Pattern 11 — PRML + Sigstore for execution integrity](https://github.com/studio-11-co/falsify-cookbook/blob/main/patterns/11-sigstore-execution.md) — closes the §8.1 gap with cosign + Rekor

falsify_inspect/__init__.py

@@ -26,17 +26,28 @@
 from falsify_inspect.core import (
     preregister,
     verify_eval_log,
+    verify_live,
+    verify_observation,
+    load_committed_manifest,
     extract_manifest_from_log,
     InspectManifest,
     PRMLVerificationError,
     MalformedLogError,
 )
 
-__version__ = "0.1.1"
+# The Inspect hook (falsify_inspect.hooks.FalsifyHooks) is intentionally NOT
+# imported here: it requires inspect_ai, and the manual API above must stay
+# importable without it. Inspect loads the hook via the `inspect_ai` entry
+# point (falsify_inspect._registry).
+
+__version__ = "0.2.0"
 
 __all__ = [
     "preregister",
     "verify_eval_log",
+    "verify_live",
+    "verify_observation",
+    "load_committed_manifest",
     "extract_manifest_from_log",
     "InspectManifest",
     "PRMLVerificationError",

falsify_inspect/_registry.py

@@ -0,0 +1,8 @@
+"""Inspect AI extension registry.
+
+Inspect discovers extensions via the ``inspect_ai`` setuptools entry point,
+which points here. Importing the hook class is enough to register it (the
+``@hooks`` decorator does the registration on import).
+"""
+
+from falsify_inspect.hooks import FalsifyHooks  # noqa: F401

falsify_inspect/core.py

@@ -275,3 +275,171 @@ def verify_eval_log(
         "actual_hash": actual_hash,
         "manifest": asdict(manifest),
     }
+
+
+# -- Hook support (in-memory verification) -------------------------------------
+#
+# These helpers exist so the Inspect hook (falsify_inspect.hooks) can verify a
+# live EvalLog object without writing it to disk first. verify_eval_log above is
+# kept untouched (it is the published, file-based path); the logic below is
+# additive.
+
+_OPS = {
+    ">=": lambda a, b: a >= b,
+    "<=": lambda a, b: a <= b,
+    ">": lambda a, b: a > b,
+    "<": lambda a, b: a < b,
+    "==": lambda a, b: a == b,
+}
+
+
+def load_committed_manifest(path: str | Path) -> tuple[dict[str, Any], str]:
+    """Load a pre-registered ``.prml.yaml`` and return ``(fields, hash)``.
+
+    Because the file is the canonical byte form produced by
+    ``InspectManifest.to_canonical_yaml()``, ``sha256(file)`` equals the hash
+    that ``preregister()`` returned at lock time. That is the committed hash.
+    """
+    raw = Path(path).read_bytes()
+    committed_hash = "sha256:" + hashlib.sha256(raw).hexdigest()
+    fields = yaml.safe_load(raw.decode("utf-8")) or {}
+    if not isinstance(fields, dict):
+        raise MalformedLogError(
+            f"manifest at {path} did not parse to a mapping (got {type(fields).__name__})"
+        )
+    return fields, committed_hash
+
+
+def verify_observation(
+    *,
+    expected_hash: str,
+    observed_value: float | None,
+    metric: str,
+    dataset: str | None,
+    dataset_hash: str | None,
+    model_version: str | None,
+    threshold: float,
+    threshold_direction: str,
+    sample_size: int | None,
+    seed: int | None,
+    pre_registered: str,
+    inspect_task: str | None = None,
+) -> dict[str, Any]:
+    """Rebuild a manifest from identity fields + observed value, then return a
+    verdict dict with an explicit ``status`` of PASS / FAIL / TAMPERED.
+
+    TAMPERED means the rebuilt hash does not equal ``expected_hash`` (the run's
+    identity, e.g. model or dataset, does not match what was pre-registered).
+    FAIL means the hash matches but the observed value misses the threshold.
+    """
+    if threshold_direction not in _OPS:
+        raise ValueError(
+            f"threshold_direction must be one of >= <= > < ==, got {threshold_direction!r}"
+        )
+    manifest = InspectManifest(
+        metric=metric,
+        value=None,
+        dataset=dataset,
+        dataset_hash=dataset_hash,
+        model_version=model_version,
+        threshold=threshold,
+        threshold_direction=threshold_direction,
+        sample_size=sample_size,
+        seed=seed,
+        pre_registered=pre_registered,
+        inspect_task=inspect_task,
+    )
+    actual_hash = manifest.hash()
+    hash_match = actual_hash == expected_hash
+    threshold_ok = (
+        observed_value is not None
+        and _OPS[threshold_direction](observed_value, threshold)
+    )
+    if not hash_match:
+        status = "TAMPERED"
+    elif threshold_ok:
+        status = "PASS"
+    else:
+        status = "FAIL"
+    return {
+        "ok": hash_match and threshold_ok,
+        "status": status,
+        "hash_match": hash_match,
+        "threshold_satisfied": threshold_ok,
+        "observed_value": observed_value,
+        "expected_hash": expected_hash,
+        "actual_hash": actual_hash,
+        "manifest": asdict(manifest),
+    }
+
+
+_MANIFEST_FIELDS = {
+    "metric", "dataset", "dataset_hash", "model_version", "threshold",
+    "threshold_direction", "sample_size", "seed", "pre_registered",
+    "prml_version", "inspect_task", "inspect_scorer",
+}
+
+
+def verify_live(
+    *,
+    manifest_path: str | Path,
+    observed_value: float | None,
+    live_model: str | None = None,
+    live_dataset: str | None = None,
+    live_dataset_hash: str | None = None,
+    live_task: str | None = None,
+) -> dict[str, Any]:
+    """Verify a live run against a committed manifest file.
+
+    Loads the pre-registered manifest (its committed hash is ``sha256(file)``),
+    overrides the identity fields the eval run actually used (model, dataset,
+    task) where the caller supplies them, rebuilds the manifest, and compares.
+    A mismatch means the run did not match the pre-registration (TAMPERED);
+    a match with a missed threshold is FAIL; a match that meets it is PASS.
+
+    All committed fields not supplied by the live run (dataset hash, seed,
+    sample size, scorer, threshold, timestamp) are taken from the manifest as
+    committed, so they do not falsely trigger TAMPERED when the eval log does
+    not expose them.
+    """
+    committed, committed_hash = load_committed_manifest(manifest_path)
+    fields = {k: v for k, v in committed.items() if k in _MANIFEST_FIELDS}
+    if live_model is not None:
+        fields["model_version"] = live_model
+    if live_dataset is not None:
+        fields["dataset"] = live_dataset
+    if live_dataset_hash is not None:
+        fields["dataset_hash"] = live_dataset_hash
+    if fields.get("inspect_task") is not None and live_task is not None:
+        fields["inspect_task"] = live_task
+
+    manifest = InspectManifest(value=None, **fields)
+    actual_hash = manifest.hash()
+    hash_match = actual_hash == committed_hash
+
+    direction = fields.get("threshold_direction")
+    threshold = fields.get("threshold")
+    threshold_ok = (
+        observed_value is not None
+        and direction in _OPS
+        and threshold is not None
+        and _OPS[direction](observed_value, threshold)
+    )
+    if not hash_match:
+        status = "TAMPERED"
+    elif threshold_ok:
+        status = "PASS"
+    else:
+        status = "FAIL"
+    return {
+        "ok": hash_match and threshold_ok,
+        "status": status,
+        "hash_match": hash_match,
+        "threshold_satisfied": threshold_ok,
+        "observed_value": observed_value,
+        "threshold": threshold,
+        "threshold_direction": direction,
+        "metric": fields.get("metric"),
+        "expected_hash": committed_hash,
+        "actual_hash": actual_hash,
+    }