Skip to content

Add Inspect AI evaluation package - #2

Merged
emmepra merged 2 commits into
devfrom
feat/1-inspect-aisi-integration
May 13, 2026
Merged

Add Inspect AI evaluation package#2
emmepra merged 2 commits into
devfrom
feat/1-inspect-aisi-integration

Conversation

@emmepra

@emmepra emmepra commented May 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an installable Inspect AI package for AHB, including dataset mapping, the ahb @task entrypoint, the public safe/unsafe scorer, tests, README usage for OpenAI/OpenRouter providers, CI test coverage, and ignored Inspect log outputs.

Why

AHB needs an upstream Inspect-ready package before it can be submitted through the Inspect Evals Register. This keeps icaro-lab/ahb as the canonical public repo and avoids a separate ahb-inspect repository.

Linked Issue

Closes #1

Verification

  • Local: uv sync --python 3.11 --extra test; uv run python -m compileall ahb_eval scripts src tests; uv run pytest; uv run inspect eval ahb_inspect/ahb --model mockllm/model --model-role grader=mockllm/model --limit 1 --no-log-realtime; uv run inspect eval ahb_inspect/ahb --model openrouter/openai/gpt-oss-20b --model-role grader=openrouter/openai/gpt-oss-20b --limit 1 --max-tokens 128 -T max_judge_tokens=128 --no-log-realtime (attack_success_rate=0.000)
  • Browser: Not applicable; no frontend behavior changed.
  • Browser evidence: Not applicable.
  • CI: Previously passed on PR Add Inspect AI evaluation package #2; pending refresh after follow-up provider dependency commit.

Auto-Merge Eligibility

Not requested. This is a normal PR to dev; local checks and one real OpenRouter smoke pass, and the PR is left open for user review before merge.

Blueprint Updates

No versioned blueprint update. The repo has no .project/blueprint and .project remains local-only for this public repo; the change follows AGENTS.md public release constraints.

Risks / Rollback

The scorer depends on current inspect-ai APIs and is pinned at inspect-ai>=0.3.220. OpenAI-compatible providers require openai>=1.50.0, now included. Rollback is a normal revert of this package layer; no dataset publication or private methodology is changed.

Follow-ups

After merge, replace the placeholder SHA in the local Register YAML draft and open a separate PR to UKGovernmentBEIS/inspect_evals under register/ahb/eval.yaml. Larger paid model validation can be run only after explicit approval.

@emmepra
emmepra merged commit 65ab433 into dev May 13, 2026
1 check passed
@emmepra
emmepra deleted the feat/1-inspect-aisi-integration branch May 13, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Inspect-native AHB evaluation package

1 participant