[DRAFT] FEAT Normalize simulated conversation prompt sources - #2503
Draft
hannahwestra25 wants to merge 3 commits into
Draft
[DRAFT] FEAT Normalize simulated conversation prompt sources#2503hannahwestra25 wants to merge 3 commits into
hannahwestra25 wants to merge 3 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7ec4a655-ad5d-481e-93a0-b5b2d1384191
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Work on #2494 highlighted a gap in the shared factory API: benchmark- or scenario-owned adversarial prompts may already exist as
SeedPromptobjects, butwith_simulated_conversation()only had a path-shaped input. That forced callers to keep prompt files coupled to the shared technique catalog or add one-off adaptation around the benchmark.This PR pulls that general capability out as an independent change based directly on
main. The preferredadversarial_chat_system_promptinput now accepts either an inlineSeedPromptor an explicitpathlib.Path. The factory resolves either form immediately and stores the canonical inline prompt, so #2494 and similar callers can own prompt content without moving prompt packaging into executors or changing persistence semantics. Plain strings are intentionally not accepted by the preferred input because non-path prompt APIs use strings for inline text; legacy string paths remain supported throughadversarial_chat_system_prompt_path.The old path-named parameter remains a silent compatibility alias for this first step, and direct or persisted legacy path-backed
SeedSimulatedConversationrecords still execute normally. The other simulated-conversation*_system_prompt_pathinputs stay unchanged here because generalizing them requires separate model, persistence, and execution compatibility work. This PR does not depend on #2494 or changeAdversarialBenchmarkitself.Tests and Documentation