Skip to content

Commit d00f673

Browse files
rlundeen2Copilot
andcommitted
FIX: Aligning scenarios with technique registry
Drop the redundant `core`-tag gates from RapidResponse and AdversarialBenchmark so scenarios expose whatever techniques the active initializer has registered, rather than re-narrowing to `core`. - rapid_response: remove `available=TagQuery.all('core')` — the pool is now every registered factory (matches the leakage pattern and the published blog, which lists the extra-only `pair` technique). - benchmark: drop the `'core' in technique_tags` clause but keep the genuine capability filters (`uses_adversarial` and `adversarial_chat is None`), so extra adversarial techniques such as `pair`/`violent_durian` can participate in the model sweep. - Update the benchmark test snapshot to mirror the production predicate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
1 parent ec6a939 commit d00f673

3 files changed

Lines changed: 12 additions & 10 deletions

File tree

pyrit/scenario/scenarios/airt/rapid_response.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,9 @@ def _build_rapid_response_technique() -> type[ScenarioTechnique]:
3535
"""
3636
Build the RapidResponse technique class dynamically from the registered factories.
3737
38-
Reads the singleton ``AttackTechniqueRegistry`` and filters to factories
39-
tagged ``core``.
38+
Reads every technique registered in the singleton ``AttackTechniqueRegistry``
39+
and exposes all of them. Which techniques are available is decided by the
40+
active initializer (the registration gate), not narrowed again here.
4041
4142
Returns:
4243
type[ScenarioTechnique]: The dynamically generated technique enum class.
@@ -50,7 +51,6 @@ def _build_rapid_response_technique() -> type[ScenarioTechnique]:
5051
return AttackTechniqueRegistry.build_technique_class_from_factories( # type: ignore[ty:invalid-return-type]
5152
class_name="RapidResponseTechnique",
5253
factories=factories,
53-
available=TagQuery.all("core"),
5454
aggregate_tags={
5555
"single_turn": TagQuery.any_of("single_turn"),
5656
"multi_turn": TagQuery.any_of("multi_turn"),

pyrit/scenario/scenarios/benchmark/adversarial.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,12 +35,14 @@ def _build_benchmark_technique() -> type[ScenarioTechnique]:
3535
"""
3636
Build the ``BenchmarkTechnique`` enum from the registered factory catalog.
3737
38-
Reads ``core`` adversarial-capable factories from the
38+
Reads adversarial-capable factories from the
3939
``AttackTechniqueRegistry`` singleton and passes them to
4040
``build_technique_class_from_factories``. Factories that bake their own
4141
``adversarial_chat`` are excluded — the benchmark sweeps each technique
4242
across the user-supplied targets, which is incompatible with a technique
43-
that pins its own adversarial target. The resulting enum has one
43+
that pins its own adversarial target. Which techniques are registered is
44+
decided by the active initializer (the registration gate); this scenario
45+
does not narrow the pool further by group. The resulting enum has one
4446
concrete member per factory (e.g. ``red_teaming``, ``tap``,
4547
``crescendo_simulated``) plus ``default`` / ``light`` / ``single_turn``
4648
/ ``multi_turn`` aggregates derived from each factory's ``technique_tags``.
@@ -56,7 +58,7 @@ def _build_benchmark_technique() -> type[ScenarioTechnique]:
5658
factories = [
5759
factory
5860
for factory in registry.get_factories_or_raise().values()
59-
if factory.uses_adversarial and "core" in factory.technique_tags and factory.adversarial_chat is None
61+
if factory.uses_adversarial and factory.adversarial_chat is None
6062
]
6163
return AttackTechniqueRegistry.build_technique_class_from_factories( # type: ignore[ty:invalid-return-type]
6264
class_name="BenchmarkTechnique",
@@ -82,7 +84,7 @@ class AdversarialBenchmark(Scenario):
8284
8385
At run time, ``_build_atomic_attacks_async`` performs the
8486
``(technique × adversarial_target × dataset)`` cross-product: for each
85-
selected adversarial-capable ``core`` factory in the
87+
selected adversarial-capable factory in the
8688
``AttackTechniqueRegistry`` and each requested target, it calls
8789
``factory.create(adversarial_chat=...)`` with the
8890
resolved target — no global registry mutation. The resulting

tests/unit/scenario/benchmark/test_adversarial.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
These tests cover the new contract:
1616
* Class metadata (VERSION, BASELINE policy, defaults).
1717
* Technique enum is built from registered factories with ``uses_adversarial=True``
18-
and the ``core`` technique tag; ``light`` aggregate preserves the
18+
that do not bake their own ``adversarial_chat``; ``light`` aggregate preserves the
1919
source ``light`` tag (excludes ``tap`` / ``crescendo_simulated``).
2020
* ``supported_parameters`` declares ``adversarial_targets: list[str]``.
2121
* ``_resolve_adversarial_targets`` raises with available names on typos.
@@ -76,7 +76,7 @@ def _build_benchmarkable_factories_snapshot() -> list:
7676
factories = build_technique_factories()
7777
finally:
7878
TargetRegistry.reset_registry_singleton()
79-
return [f for f in factories if f.uses_adversarial and "core" in f.technique_tags]
79+
return [f for f in factories if f.uses_adversarial and f.adversarial_chat is None]
8080

8181

8282
_BENCHMARKABLE_FACTORIES = _build_benchmarkable_factories_snapshot()
@@ -202,7 +202,7 @@ class TestAdversarialBenchmarkTechnique:
202202
"""Tests for ``_build_benchmark_technique`` using the registry-based factory API."""
203203

204204
def test_technique_built_from_registered_adversarial_factories(self):
205-
"""Each registered ``core`` adversarial factory produces one concrete enum member."""
205+
"""Each registered adversarial factory produces one concrete enum member."""
206206
technique_cls = _build_benchmark_technique()
207207
aggregate_names = {"all"} | technique_cls.get_aggregate_tags()
208208
concrete_members = [m for m in technique_cls if m.value not in aggregate_names]

0 commit comments

Comments
 (0)