Skip to content

[DRAFT] FEAT Normalize simulated conversation prompt sources - #2503

Draft
hannahwestra25 wants to merge 3 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-normalize-simulated-prompt-source
Draft

[DRAFT] FEAT Normalize simulated conversation prompt sources#2503
hannahwestra25 wants to merge 3 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-normalize-simulated-prompt-source

Conversation

@hannahwestra25

@hannahwestra25 hannahwestra25 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Description

Work on #2494 highlighted a gap in the shared factory API: benchmark- or scenario-owned adversarial prompts may already exist as SeedPrompt objects, but with_simulated_conversation() only had a path-shaped input. That forced callers to keep prompt files coupled to the shared technique catalog or add one-off adaptation around the benchmark.

This PR pulls that general capability out as an independent change based directly on main. The preferred adversarial_chat_system_prompt input now accepts either an inline SeedPrompt or an explicit pathlib.Path. The factory resolves either form immediately and stores the canonical inline prompt, so #2494 and similar callers can own prompt content without moving prompt packaging into executors or changing persistence semantics. Plain strings are intentionally not accepted by the preferred input because non-path prompt APIs use strings for inline text; legacy string paths remain supported through adversarial_chat_system_prompt_path.

The old path-named parameter remains a silent compatibility alias for this first step, and direct or persisted legacy path-backed SeedSimulatedConversation records still execute normally. The other simulated-conversation *_system_prompt_path inputs stay unchanged here because generalizing them requires separate model, persistence, and execution compatibility work. This PR does not depend on #2494 or change AdversarialBenchmark itself.

Tests and Documentation

  • 397 focused and adjacent unit tests passed
  • Ruff check/format, ty, async-suffix, compile, and diff checks passed
  • Jupytext paired sources and round-trip checks passed
  • Updated the attack-techniques documentation pair

hannahwestra25 and others added 3 commits August 26, 2026 13:36
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 7ec4a655-ad5d-481e-93a0-b5b2d1384191
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant