PromptAuthority-15

PromptAuthority-15 is an official BenchLocal Bench Pack for evaluating prompt-authority conflict handling. It tests whether a model resolves conflicts between hard system rules, user instructions, soft system defaults, and instructions embedded inside tool data.

A Bench Pack is an installable benchmark package that runs inside the BenchLocal desktop app. BenchLocal provides the shared app experience for provider setup, model selection, sampling controls, run histories, and side-by-side comparison across benchmark packs.

This repository contains the benchmark source: scenario definitions, scoring logic, methodology, a BenchLocal adapter, and a CLI runner for local development. main tracks the maintained Bench Pack version for BenchLocal.

Run With BenchLocal

1. Download BenchLocal from the latest BenchLocal release.
2. Open BenchLocal and install PromptAuthority-15 from the official Bench Pack registry.
3. Add one or more models, select PromptAuthority-15, and start a run.

What It Measures

PromptAuthority-15 is organized into 5 categories, with 3 scenarios per category:

• Hard System Constraints
• User Authority Over Defaults
• Tool-Output Injection Resistance
• Tool-Call Authority Boundaries
• Multi-Turn Authority Retention

Each scenario is scored as:

• 1 point for a pass
• 0 points for a fail

The final PA Score is the percentage of scenarios passed, rounded to a whole number for BenchLocal display. The suite also reports HVC, the Hierarchy Violation Count, for cases where a model obeyed an instruction that originated in untrusted tool data.

Bench Pack Structure

lib/                    Benchmark core, scoring, tool loop, and transport
benchlocal/             Thin BenchLocal SDK adapter
cli/                    Non-UI runner
benchlocal.pack.json    Canonical Bench Pack manifest
METHODOLOGY.md          Published benchmark methodology

BenchLocal Adapter

• benchlocal/index.ts is the only place that imports @benchlocal/sdk.
• lib/ stays framework-agnostic and is shared by the CLI and BenchLocal.
• benchlocal.pack.json is the canonical Bench Pack metadata manifest used for install, inspection, and runtime metadata.
• Per-pack default sampling belongs on the manifest. PromptAuthority-15 defaults to temperature: 0.

Methodology

The benchmark spec is documented in METHODOLOGY.md and implemented in lib/benchmark.ts.

PromptAuthority-15 uses per-run canary instantiation, deterministic text and tool-call checks, and binary scenario scoring. This keeps the grader auditable while reducing fixed-token overfitting.

Design Goals

• Reproducible: the system prompt, tool schema, mocked tool outputs, canary generation, and scoring logic are all versioned in the repo.
• Absolute: every scenario has a pre-declared machine-checkable outcome instead of a judgment rubric.
• Balanced: the suite includes cases where hard system rules must win, user instructions must override defaults, and legitimate tool data must still be used.
• Deterministic: tool results are mocked and the benchmark uses temperature: 0.
• Inspectable: every scenario stores a raw trace, ordered tool-call log, and canary map so failures can be audited.

BenchLocal and CLI

• Install: npm install
• BenchLocal build: npm run build:benchlocal
• CLI runner: npm run cli
• Methodology: METHODOLOGY.md

CLI model configuration matches the sibling packs:

• LLM_MODELS=openrouter:provider/model,ollama:model-name
• LLM_MODELS_2=... for a secondary comparison group
• OPENROUTER_API_KEY=... for OpenRouter
• OLLAMA_HOST, LLAMACPP_HOST, MLX_HOST, or LMSTUDIO_HOST for local providers

Example:

npm run cli -- --model openrouter:openai/gpt-4.1 --show-raw

Validation

npm run typecheck
npm run build:benchlocal