RFC: Port the evidence-led agent red-team benchmark into AGT's testing suite

[kerberosmansour]

Proposal

Should AGT add a completed, evidence-led benchmark to its existing red-team testing suite?

The reference implementation is already built and running in kerberosmansour/AGT-Embeddings-Experiment. Its latest completed release gate joined 54,034 static detector records with 250 live, sandboxed behavioural records. It also includes a 24-scenario CI smoke suite and a balanced 240-scenario measurement suite. The capability is documented in issue #41 and the completed consolidation is in PR #40.

AGT already has useful red-team coverage. This proposal would add to it, not replace it. The main difference is that the benchmark follows an attack beyond the initial verdict: it records what the agent attempted, whether the action executed, and which control stopped it.

At a glance

Area	AGT today	Completed reference benchmark
Prompt defences	17 static defence vectors	Uses detector results as one evidence source, then joins them to action outcomes
Adversarial playbooks	5 built-in playbooks containing 13 attack steps	24-scenario smoke suite plus 240-scenario balanced measurement suite
Policy regression	28 scenarios run against 3 starter packs (84 policy checks)	Six cross-layer trap classes, including attack, evasion, hard-benign, and near-miss cases
Corpus-scale measurement	Existing suites are focused on targeted regression and resilience testing	54,034 labelled static records in the completed release run
Agent behaviour	Playbook resilience and policy decisions	Deterministic mock traces plus 250 live sandboxed records in the completed release run
Reporting	Grades, pass rates, resilience scores, and policy outcomes	Detection-to-action matrix, containment, attack success, false positives, and utility preservation

These numbers are not a head-to-head score: the suites do different jobs. AGT's current tests provide focused product regression coverage. The reference benchmark adds broader measurement, explicit benign controls, and action-level evidence.

Feedback requested

Would maintainers and contributors find this useful as part of AGT? In particular:

• Should the completed capability be ported into AGT's testing suite?
• Is tests/redteam plus agent-sre/chaos the right ownership boundary?
• Should an initial contribution include only the deterministic smoke and mock-behavioural tiers, with the larger corpus and live tier following separately?
• Which of the six trap classes would be most valuable to AGT users?

Why this may be useful

An attack being detected does not necessarily mean it was contained. A tool action can still execute after a warning, while a downstream control can sometimes stop an attack that the detector missed. The benchmark makes both cases visible by recording the detection verdict and action result for the same scenario:

	Action blocked / contained	Action executed
Detected	Expected result	High-severity control failure
Undetected	Defence in depth held	Worst-case outcome

This gives maintainers a clearer view of where a control worked, where it failed, and what should be fixed next. It also checks whether benign work still succeeds, so blocking everything cannot produce a strong result.

Threat coverage: six trap classes

Trap class	Attack surface	Examples
Content Injection	Content the agent reads	Hidden HTML comments, poisoned tool output, human-sees-X / agent-reads-Y format tricks
Semantic Manipulation	Meaning of the task	Authority spoofing, roleplay legitimacy, goal reframing, making unsafe requests sound routine
Cognitive State	Memory and session state	Poisoned memory, cross-session echoes, preference manipulation, stale context
Behavioural Control	Tool use and actions	Approval fatigue, tool-schema confusion, irreversible-pressure prompts, least-privilege blur
Systemic	The wider agent ecosystem	Cross-agent relay, MCP registry misdirection, package-name confusion, delegation loops
Human-in-the-Loop	Approval and review workflows	Fake approvals, social proof and time pressure, visible-safe / hidden-unsafe requests

Detailed Design

The reference implementation provides the following capabilities for adaptation:

1. Typed scenario, result, and tool-trace contracts

   • Each scenario identifies its trap class, target layer, delivery surface, expected behaviour, and evidence level.
   • Action traces distinguish attempted, executed, and blocked_at rather than inferring containment from model text.

2. Layered execution tiers

   Tier	Size	Purpose
   Smoke suite	24 scenarios	Fast, deterministic CI regression check
   Measurement suite	240 scenarios	Per trap class: 8 canonical attacks, 16 evasions, 8 hard-benign cases, and 8 near misses
   Static corpus tier	54,034 records in the completed run	Corpus-scale detector evidence
   Live behavioural tier	250 pre-registered records plus a benign utility arm in the completed run	Real agent behaviour inside a Linux sandbox; opt-in

3. Explicit evidence levels

   • Declared evidence.
   • Static detector evidence.
   • Deterministic mock behavioural evidence.
   • Live sandboxed behavioural evidence.

   Reports cannot promote a result above the level at which it was produced.

4. Joint reporting

   • Detection verdicts are joined to action outcomes for the same scenario.
   • Reports include canonical catch rate, evasion degradation, false-positive rate, utility preservation, action-level attack success rate, containment rate, and benign false-block rate.
   • The off-diagonal undetected → contained and detected → executed cases remain visible rather than being hidden by a single aggregate score.

5. Safety and integrity controls

   • Generated reports are metadata-only and are checked for raw payloads, secrets, live URLs, and personal data.
   • Live tests fail closed when the required sandbox cannot be established; they do not silently fall back to weaker evidence.
   • Live execution denies network egress, scrubs the environment, and prevents access to the host filesystem.
   • Reports are explicitly non-certifying and carry reproducibility hashes for the scenario set, corpus, and artifacts.

Proposed AGT integration

A staged port could preserve AGT's existing interfaces and package boundaries:

1. Add the deterministic schemas, fixtures, mock harness, and smoke runner under tests/redteam/.
2. Place reusable adversarial execution support under agent-sre/chaos, consistent with the completed package-boundary decision in AGT issue #200.
3. Optionally expose the benchmark through agt red-team once the repository-level contract is stable.
4. Keep live behavioural runs opt-in and Linux-only initially; the default CI path remains deterministic and side-effect-free.
5. Treat external corpora as pinned inputs with manifests rather than committing large or sensitive payload collections by default.

The first contribution could be limited to schemas, the 24-scenario smoke suite, mock behavioural traces, joint reporting, and artifact hygiene. The full measurement and live tiers could follow after maintainers validate placement and maintenance expectations.

Alternatives Considered

1. Keep the benchmark only in the external repository. This avoids new AGT maintenance cost but fragments the testing story and prevents AGT's own controls from using the richer action-level evidence in CI.
2. Port only the additional scenarios. This increases attack coverage but loses the central distinction between detection, attempted action, containment, and execution.
3. Extend only the existing resilience score. A single score is simpler, but it can hide the high-severity detected → executed case and benign utility failures.
4. Import the complete corpus into AGT. This is easy to reproduce locally but increases repository size and payload-handling risk. Pinned external inputs plus manifests are a safer default.

Security Implications

This proposal expands security testing without changing AGT's runtime trust model. The main risks are test payload leakage, unintended live side effects, false confidence from weak evidence, and test-set tuning.

The proposed controls are:

• Metadata-only, raw-free report artifacts with a fail-closed hygiene check.
• No secrets, personal data, or live targets in committed scenarios or generated reports.
• OS-enforced isolation for live behavioural tests, including denied network egress, scrubbed environment variables, and no host filesystem access.
• Refusal to claim live evidence when the sandbox or required execution path is unavailable.
• Pre-registered live samples and frozen thresholds before results are reviewed.
• A permanent non-certification statement in machine-readable and human-readable reports.

Migration / Backward Compatibility

The port can be additive:

• Existing agt red-team commands and playbooks remain unchanged.
• Existing tests/redteam cases continue to run.
• The deterministic smoke tier can be introduced as a separate CI job.
• Live testing remains opt-in and does not affect contributors without the required Linux sandbox.
• Public CLI changes, if desired, can follow as a separate reviewed change after the test contracts stabilize.

Scope

Cross-package (2-3 packages): repository-level tests, agent-sre, and optionally agent-compliance for CLI exposure.

Target Placement

Existing tests/redteam for the benchmark contract and CI suite, with reusable adversarial execution support in agent-sre/chaos. Optional public CLI integration can remain a later decision.

Prior Art

• Completed reference implementation: kerberosmansour/AGT-Embeddings-Experiment
• Capability overview and usage: kerberosmansour/AGT-Embeddings-Experiment#41
• Completed implementation: kerberosmansour/AGT-Embeddings-Experiment#40
• Existing AGT red-team guide: tests/redteam/README.md
• Existing AGT red-team CLI tutorial: docs/tutorials/47-red-team-testing.md
• Prior AGT package-boundary decision: microsoft/agent-governance-toolkit#200

Checklist

• I have searched existing issues and RFCs for duplicates.
• I have read the ADR index for related decisions.
• I am willing to implement this RFC or help review an implementation.

[github-actions]

🟡 Contributor Check: MEDIUM

Check	Result
Profile	MEDIUM
Credential	MEDIUM
Overall	MEDIUM

Automated check by AGT Contributor Check.

[imran-siddique]

Supportive of this. Following an attack past the detection verdict to whether the action actually executed and which control stopped it is exactly the distinction AGT is built on: a detected attack is not a contained one, and deterministic enforcement is what makes the difference. A benchmark that measures containment and not just detection reinforces that thesis rather than duplicating existing coverage, so it earns its place.

Answering your four questions directly.

1. Port it? Yes, additively. Keep the detection-to-action matrix (detected/executed as the high-severity cell, undetected/contained as defense-in-depth) as the centerpiece; that is the part with no equivalent in the current suite. Alternatives 2 and 3 (scenarios-only, or folding into a single resilience score) lose exactly that signal, so I would not take those.

2. Ownership boundary. tests/redteam for the benchmark contract and CI suite is right. Reusable adversarial execution in agent-sre/chaos is a reasonable home for the live tier. I would hold agent-compliance CLI exposure as a later, separate decision after the test contracts stabilize, rather than landing it in the first PR. That keeps the initial surface small and the public CLI stable, consistent with #200.

3. Phasing. Yes. First contribution: the deterministic smoke tier plus the mock-behavioural tier as a separate CI job, additive to the existing agt red-team path. The 240-scenario measurement suite, the 54k static corpus, and the 250-record live tier follow as separate reviewed changes. On the corpus specifically, keep it external and pinned (your alternative 4, importing the full corpus in-tree, is the one to avoid) so we do not carry payload size and handling risk in the repo. Manifests plus pinned inputs over a vendored corpus.

4. Trap classes. Most valuable first: Content Injection (indirect injection via tool output and read content maps directly to the MCP security scanner and the I/O-handling boundary, where the detected-versus-executed gap is most instructive), plus the hard-benign and near-miss classes. The benign controls matter as much as the attacks: a security suite has to prove it preserves utility, otherwise "block everything" scores well and teaches nothing. Semantic Manipulation is a good third once the contract is proven.

On the security controls you list, I would treat them as requirements, not options: metadata-only artifacts with a fail-closed hygiene check, OS-enforced isolation with denied egress and scrubbed env for any live tier, explicit refusal to claim live evidence when the sandbox is unavailable, and the permanent non-certification statement in both machine and human-readable reports. Those are the right defaults for this repo.

Suggested next step: open an initial PR scoped to the deterministic smoke and mock-behavioural tiers plus the benchmark contract under tests/redteam, corpus and live tier deferred. Happy to review the contract design first if you want to align on the report schema before building it out.

[kerberosmansour]

Implementation update:

• PR feat(redteam): add detection-to-action smoke benchmark #3362 now contains the accepted deterministic L2 scope plus additional input-validation, Windows/Bash portability, and CI hardening.
• The reference implementation fixes are in fix(redteam): fail closed on incomplete live traces kerberosmansour/AGT-Embeddings-Experiment#42. That PR distinguishes a genuine observed refusal from incomplete trace capture, removes import-time fixture loading, and makes Bash/corpus-backed tests skip with explicit setup guidance.

The AGT contribution remains intentionally L2-only; the 240-scenario measurement suite, external corpus tier, and live sandboxed tier are still deferred for separate review. These follow-up changes preserve the detection-to-action matrix while ensuring incomplete evidence cannot appear as containment.