benchmarks

ZIRAN Benchmark Coverage

How ZIRAN's attack vector library compares against published AI agent security benchmarks.

Current State

Metric	Value
Attack vectors	639
Attack categories	11
OWASP LLM Top 10	100.0% (10/10)
Multi-turn tactics	10
Encoding types	12
Benchmarks analyzed	20
MITRE ATLAS techniques covered	72/86 (14/14 agent-specific)
Gap closure	39.1% (9/23)

OWASP LLM Top 10 Coverage

Code	Category	Vectors	Status
LLM01	Prompt Injection	468	✅ Comprehensive
LLM02	Insecure Output Handling	201	✅ Comprehensive
LLM03	Training Data Poisoning	19	✅ Strong
LLM04	Model Denial of Service	14	✅ Strong
LLM05	Supply Chain Vulnerabilities	18	✅ Strong
LLM06	Sensitive Information Disclosure	110	✅ Comprehensive
LLM07	Insecure Plugin Design	145	✅ Comprehensive
LLM08	Excessive Agency	151	✅ Comprehensive
LLM09	Overreliance	15	✅ Strong
LLM10	Unbounded Consumption	10	✅ Strong

Benchmark Comparison

Benchmark	Venue	Dimension	Target	ZIRAN	Progress	Status	Gap
AgentHarm	ICLR 2025	Harm categories	11	11	███████████████ 100.0%	✅ closed	GAP-06
AgentHarm	ICLR 2025	Multi-step vectors	440	161	█████░░░░░░░░░░ 36.6%	🚧 open	GAP-23
InjecAgent	ACL 2024	Indirect injection vectors	1,054	63	█░░░░░░░░░░░░░░ 6.0%	🚧 open	GAP-02
AgentDojo	NeurIPS 2024	Indirect injection vectors	629	63	██░░░░░░░░░░░░░ 10.0%	🚧 open	GAP-02
		Utility measurement (baseline + post-attack)	1	1	███████████████ 100.0%
HarmBench	ICML 2024	Attack tactics	18	10	████████░░░░░░░ 55.6%	✅ closed	GAP-08
		Jailbreak vectors	510	206	██████░░░░░░░░░ 40.4%
JailbreakBench	NeurIPS 2024	JBB categories (10)	10	10	███████████████ 100.0%	✅ closed	GAP-15
		Prompt injection vectors	100	206	███████████████ 100%
StrongREJECT	2024	StrongREJECT composite formula	1	1	███████████████ 100.0%	✅ closed	GAP-04
		Scoring dimensions (refusal, specificity, convincingness)	3	3	███████████████ 100.0%
MCPTox	2025	MCP vectors	1,312	101	█░░░░░░░░░░░░░░ 7.7%	🚧 open	GAP-03
Agent Security Bench (ASB)	2024	Attack categories	10	11	███████████████ 100%	🚧 open	GAP-01
		Total vectors	400	639	███████████████ 100%
		Utility-under-attack measurement	1	1	███████████████ 100.0%
TensorTrust	2024	Prompt injection vectors	126,000	206	░░░░░░░░░░░░░░░ 0.2%	🚧 open	GAP-16
		Representative pattern families	—	11	Distinct TensorTrust pattern families covered
WildJailbreak	2024	Jailbreak tactics	105,000	11	░░░░░░░░░░░░░░░ 0.0%	🚧 open	GAP-17
		WildJailbreak-inspired multi-turn vectors	—	10	Distinct tactic families from WildJailbreak
LLMail-Inject / RAG Poisoning	2024	RAG retrieval-targeted vectors	—	13	Retrieval-ranked payloads across multiple document framings	🚧 open	GAP-13
Agent-SafetyBench	2024	Business impact types	8	7	█████████████░░ 87.5%	🚧 open	GAP-07
BIPIA	2024	Indirect injection vectors	—	63	Multi-domain benchmark — no fixed target count	🚧 open	GAP-02
CyberSecEval	Meta, 2024	Code-generation safety vectors	—	10	Code-gen safety + cyber knowledge elicitation families	🚧 open	GAP-18
		Total library overlap	—	639	Multi-category benchmark — partial overlap
ToolEmu	2024	Tool manipulation vectors	144	176	███████████████ 100%	🚧 open	GAP-19
		Dedicated sandbox-evasion vectors	—	10	Sandbox-evasion vectors distinct from generic tool manipulation
R-Judge	2024	R-Judge risk types (10)	10	10	███████████████ 100.0%	✅ closed	GAP-20
		Risk scoring detectors	—	5	5 detectors — different approach than interaction records
AILuminate	MLCommons, 2025	Resilience gap metric	1	1	███████████████ 100.0%	✅ closed	GAP-09
		Baseline performance measurement	1	1	███████████████ 100.0%
		Under-attack performance measurement	1	1	███████████████ 100.0%
ALERT	2024	ALERT micro categories (32)	32	32	███████████████ 100.0%	✅ closed	GAP-21
		Harm categories	—	11	N/A
MITRE ATLAS	MITRE, 2025	ATLAS tactics covered	16	16	███████████████ 100.0%	🚧 open	GAP-22
		ATLAS techniques mapped	86	72	█████████████░░ 83.7%
		Agent-specific techniques covered	14	14	███████████████ 100.0%

Gap Status

ID	Gap	Priority	Status
GAP-01	Benchmark harness	critical	🚧 open (#32)
GAP-02	Indirect prompt injection scale	critical	🚧 open (#33)
GAP-03	MCP tool poisoning	critical	🚧 open (#34)
GAP-04	Quality-aware jailbreak scoring	critical	✅ closed (#35)
GAP-05	Utility-under-attack measurement	important	✅ closed (#36)
GAP-06	Harmful multi-step task testing	important	✅ closed (#37)
GAP-07	Business impact categorization	important	🚧 open (#38)
GAP-08	Jailbreak tactic breadth	important	✅ closed (#39)
GAP-09	Resilience gap metric	important	✅ closed (#40)
GAP-10	OWASP LLM04 (Model DoS)	lower	✅ closed (#41)
GAP-11	OWASP LLM05 (Supply Chain)	lower	🚧 open (#42)
GAP-12	OWASP LLM10 (Model Theft)	lower	🚧 open (#43)
GAP-13	RAG-specific poisoning	lower	🚧 open (#44)
GAP-14	Defense evasion measurement	lower	🚧 open (#45)
GAP-15	JailbreakBench coverage	lower	✅ closed (#54)
GAP-16	TensorTrust coverage	lower	🚧 open (#55)
GAP-17	WildJailbreak coverage	lower	🚧 open (#56)
GAP-18	CyberSecEval coverage	lower	🚧 open (#57)
GAP-19	ToolEmu coverage	lower	🚧 open (#58)
GAP-20	R-Judge coverage	lower	✅ closed (#59)
GAP-21	ALERT coverage	lower	✅ closed (#60)
GAP-22	MITRE ATLAS technique mapping	important	🚧 open (#61)
GAP-23	AgentHarm multi-step vector scale	important	🚧 open (#131)

Vector Inventory

By Attack Category

Category	Vectors
prompt_injection	206
tool_manipulation	176
indirect_injection	63
data_exfiltration	56
privilege_escalation	35
system_prompt_extraction	27
memory_poisoning	20
authorization_bypass	17
chain_of_thought_manipulation	15
model_dos	13
multi_agent	11

By Tactic

Tactic	Vectors
single	410
context_buildup	63
crescendo	38
persona_shift	23
hypothetical	17
role_play	16
distraction	15
refusal_suppression	15
code_mode	14
few_shot	14
language_switch	14

By Severity

Severity	Vectors
critical	367
high	201
low	1
medium	70

Scripts

Each script is independently runnable:

# Individual scripts
uv run python benchmarks/inventory.py
uv run python benchmarks/owasp_coverage.py
uv run python benchmarks/benchmark_comparison.py
uv run python benchmarks/gap_status.py

# Generate all results + markdown report
uv run python benchmarks/generate_all.py

# Write JSON output
uv run python benchmarks/inventory.py --json benchmarks/results/inventory.json

Regenerating

After adding new vectors or closing gaps, regenerate:

uv run python benchmarks/generate_all.py

This updates benchmarks/results/*.json, benchmarks/README.md, and docs/reference/benchmarks/coverage-comparison.md.