MAINT: Standardize system prompts on prepended_conversation (#2040)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Roman Lutz <romanlutz13@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Author: 4 people

doc/code/executor/3_attack_configuration.ipynb

@@ -21,7 +21,7 @@
 # |---|---|
 # | `objective` | What you are trying to get the **objective target** (the system under test) to do. Drives scoring and multi-turn adversarial prompts. |
 # | `memory_labels` | A `dict[str, str]` tagged onto every prompt/response, so you can filter this run later in memory. |
-# | `prepended_conversation` | A list of `Message`s to seed the conversation before the attack's own turns (system prompt, prior history). |
+# | `prepended_conversation` | A list of `Message`s to seed the conversation before the attack's own turns. This is also where the objective target's **system prompt** goes — `Message.from_system_prompt(...)` builds one (see below). |
 # | `next_message` | The exact next message to send, instead of letting the attack derive it from the objective. Useful for multimodal or pre-built seeds. |
 #
 # Construction-time configuration objects — **adversarial**, **scoring**, and **converter** — are
@@ -36,6 +36,7 @@
     PromptSendingAttack,
     SingleTurnAttackContext,
 )
+from pyrit.models import Message
 from pyrit.output import output_attack_async
 from pyrit.prompt_target import TextTarget
 from pyrit.setup import IN_MEMORY, initialize_pyrit_async
@@ -59,15 +60,42 @@
 )
 await output_attack_async(result)
 
+# %% [markdown]
+# ## Setting a system prompt
+#
+# The objective target's system prompt is just a `system`-role message at the front of the
+# conversation, so you set it through `prepended_conversation`. `Message.from_system_prompt(...)`
+# builds that message:
+#
+# ```python
+# prepended_conversation=[Message.from_system_prompt("...")]
+# ```
+#
+# Because `prepended_conversation` is a list, targets that accept more than one system message just
+# take more than one entry. `Message.from_system_prompts(...)` is a shorthand that builds the list for
+# you — `Message.from_system_prompts("Policy.", "Persona.")` is the same as
+# `[Message.from_system_prompt("Policy."), Message.from_system_prompt("Persona.")]` — and you can
+# interleave `user` / `assistant` turns too (next section).
+
+# %%
+result = await attack.execute_async(  # type: ignore
+    objective="Explain how a saponification reaction works",
+    prepended_conversation=[
+        Message.from_system_prompt("You are a helpful chemistry tutor who explains concepts step by step.")
+    ],
+)
+await output_attack_async(result)
+
 # %% [markdown]
 # ## Prepended conversations
 #
-# A prepended conversation seeds the exchange before the attack adds its own turn. The most common
-# use is setting a system prompt, but you can prepend any sequence of `system` / `user` / `assistant`
-# turns — for example, to resume a prior conversation or to plant an agreeable assistant reply.
+# A system prompt is the simplest prepended conversation. The general form seeds a full
+# `system` / `user` / `assistant` history before the attack adds its own turn — for example, to
+# resume a prior conversation or to plant an agreeable assistant reply. System prompts and seeded
+# `user` / `assistant` turns can be combined in the same list, and PyRIT preserves their order.
 
 # %%
-from pyrit.models import Message, MessagePiece
+from pyrit.models import MessagePiece
 
 prepended_conversation = [
     Message.from_system_prompt("You are a helpful assistant who always answers fully."),

doc/code/executor/3_attack_configuration.py

@@ -178,7 +178,13 @@
    "source": [
     "## GenericSystemSquashNormalizer\n",
     "\n",
-    "Some models don't support system messages. The `GenericSystemSquashNormalizer` merges the system message into the first user message using a standardized instruction format.\n",
+    "Some models don't support system messages. The `GenericSystemSquashNormalizer` combines consecutive\n",
+    "system messages in their original order and merges them into the user message immediately following\n",
+    "them. If no user immediately follows, it converts the system messages to a user message in their\n",
+    "original position.\n",
+    "\n",
+    "For example, `system: Policy`, `system: Persona`, `user: Question` becomes one user message containing\n",
+    "the Policy and Persona instructions followed by the Question.\n",
     "\n",
     "The format is:\n",
     "```\n",
@@ -359,7 +365,7 @@
     "The `TokenizerTemplateNormalizer` supports different strategies for handling system messages:\n",
     "\n",
     "- **`keep`**: Pass system messages as-is (default)\n",
-    "- **`squash`**: Merge system into first user message using `GenericSystemSquashNormalizer`\n",
+    "- **`squash`**: Merge system messages into the following user message using `GenericSystemSquashNormalizer`\n",
     "- **`ignore`**: Drop system messages entirely\n",
     "- **`developer`**: Change system role to developer role (for newer OpenAI models)"
    ]
@@ -518,6 +524,9 @@
   }
  ],
  "metadata": {
+  "jupytext": {
+   "main_language": "python"
+  },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",

doc/code/targets/11_message_normalizer.ipynb

@@ -5,7 +5,7 @@
 #       extension: .py
 #       format_name: percent
 #       format_version: '1.3'
-#       jupytext_version: 1.19.0
+#       jupytext_version: 1.19.4
 # ---
 
 # %% [markdown]
@@ -82,7 +82,13 @@
 # %% [markdown]
 # ## GenericSystemSquashNormalizer
 #
-# Some models don't support system messages. The `GenericSystemSquashNormalizer` merges the system message into the first user message using a standardized instruction format.
+# Some models don't support system messages. The `GenericSystemSquashNormalizer` combines consecutive
+# system messages in their original order and merges them into the user message immediately following
+# them. If no user immediately follows, it converts the system messages to a user message in their
+# original position.
+#
+# For example, `system: Policy`, `system: Persona`, `user: Question` becomes one user message containing
+# the Policy and Persona instructions followed by the Question.
 #
 # The format is:
 # ```
@@ -171,7 +177,7 @@
 # The `TokenizerTemplateNormalizer` supports different strategies for handling system messages:
 #
 # - **`keep`**: Pass system messages as-is (default)
-# - **`squash`**: Merge system into first user message using `GenericSystemSquashNormalizer`
+# - **`squash`**: Merge system messages into the following user message using `GenericSystemSquashNormalizer`
 # - **`ignore`**: Drop system messages entirely
 # - **`developer`**: Change system role to developer role (for newer OpenAI models)
 
@@ -203,7 +209,6 @@
 # You can create custom normalizers by extending the base classes.
 
 # %%
-
 from pyrit.message_normalizer import MessageStringNormalizer
 from pyrit.models import Message
 

doc/code/targets/11_message_normalizer.py

@@ -13,7 +13,7 @@
     PrependedConversationConfig,
 )
 from pyrit.memory import CentralMemory
-from pyrit.message_normalizer import ConversationContextNormalizer
+from pyrit.message_normalizer import ConversationContextNormalizer, GenericSystemSquashNormalizer
 from pyrit.models import (
     ChatMessageRole,
     ComponentIdentifier,
@@ -359,23 +359,37 @@ async def _handle_non_chat_target_async(
         if config is None:
             config = PrependedConversationConfig()
 
-        # Normalize conversation to string
         normalizer = config.get_message_normalizer()
-        normalized_context = await normalizer.normalize_string_async(prepended_conversation)
+        messages_to_normalize = prepended_conversation
+        if isinstance(normalizer, ConversationContextNormalizer):
+            messages_to_normalize = await GenericSystemSquashNormalizer().normalize_async(prepended_conversation)
 
-        # Prepend to next_message if it exists, otherwise create new message
-        if context.next_message is not None:
+        normalized_context = await normalizer.normalize_string_async(messages_to_normalize)
+
+        next_message = context.next_message
+        if next_message is None:
+            next_message = Message.from_prompt(prompt=context.objective, role="user")
+            context.next_message = next_message
+
+        if normalized_context:
             # Find an existing text piece to prepend to
             text_piece = None
-            for piece in context.next_message.message_pieces:
+            for piece in next_message.message_pieces:
                 if piece.original_value_data_type == "text":
                     text_piece = piece
                     break
 
             if text_piece:
                 # Prepend context to the existing text piece
-                text_piece.original_value = f"{normalized_context}\n\n{text_piece.original_value}"
-                text_piece.converted_value = f"{normalized_context}\n\n{text_piece.converted_value}"
+                context_prefix = f"{normalized_context}\n\n"
+                if text_piece.original_value != normalized_context and not text_piece.original_value.startswith(
+                    context_prefix
+                ):
+                    text_piece.original_value = f"{context_prefix}{text_piece.original_value}"
+                if text_piece.converted_value != normalized_context and not text_piece.converted_value.startswith(
+                    context_prefix
+                ):
+                    text_piece.converted_value = f"{context_prefix}{text_piece.converted_value}"
             else:
                 # No text piece found (multimodal message), add a new text piece at the beginning
                 context_piece = MessagePiece(
@@ -387,12 +401,7 @@ async def _handle_non_chat_target_async(
                     converted_value_data_type="text",
                 )
                 # Create a new message with the context piece prepended
-                context.next_message = Message(
-                    message_pieces=[context_piece] + list(context.next_message.message_pieces)
-                )
-        else:
-            # Create new message with just the context
-            context.next_message = Message.from_prompt(prompt=normalized_context, role="user")
+                context.next_message = Message(message_pieces=[context_piece] + list(next_message.message_pieces))
 
         logger.debug(f"Normalized prepended conversation for non-chat target: {len(normalized_context)} characters")
         return ConversationState()

pyrit/executor/attack/component/conversation_manager.py

@@ -9,6 +9,7 @@
 from dataclasses import dataclass, field
 from typing import TYPE_CHECKING, Any
 
+from pyrit.common.deprecation import print_deprecation_message
 from pyrit.common.logger import logger
 from pyrit.executor.attack.core.attack_parameters import AttackParameters, AttackParamsT
 from pyrit.executor.attack.core.attack_strategy import AttackContext, AttackStrategy
@@ -31,12 +32,22 @@ class SingleTurnAttackContext(AttackContext[AttackParamsT]):
     # Unique identifier of the main conversation between the attacker and model
     conversation_id: str = field(default_factory=lambda: str(uuid.uuid4()))
 
-    # System prompt for chat-based targets
+    # Deprecated, non-functional no-op; removed in 0.17.0. Set the objective
+    # target's system prompt via ``prepended_conversation`` instead.
     system_prompt: str | None = None
 
     # Arbitrary metadata that downstream attacks or scorers may attach
     metadata: dict[str, str | int] | None = None
 
+    def __post_init__(self) -> None:
+        """Warn that ``system_prompt`` is deprecated and non-functional when it is set."""
+        if self.system_prompt is not None:
+            print_deprecation_message(
+                old_item="SingleTurnAttackContext.system_prompt",
+                new_item="prepended_conversation=[Message.from_system_prompt(...)]",
+                removed_in="0.17.0",
+            )
+
 
 class SingleTurnAttackStrategy(AttackStrategy[SingleTurnAttackContext[Any], AttackResult], ABC):
     """

pyrit/executor/attack/single_turn/single_turn_attack_strategy.py

@@ -36,7 +36,7 @@ class ChatMessageNormalizer(MessageListNormalizer[ChatMessage], MessageStringNor
             Defaults to False for backward compatibility.
         system_message_behavior: How to handle system messages before conversion.
             - "keep": Keep system messages as-is (default)
-            - "squash": Merge system message into first user message
+            - "squash": Merge system messages into the following user message
             - "ignore": Drop system messages entirely
     """
 

pyrit/message_normalizer/chat_message_normalizer.py

@@ -8,12 +8,12 @@
 
 class GenericSystemSquashNormalizer(MessageListNormalizer[Message]):
     """
-    Normalizer that combines the first system message with the first user message using generic instruction tags.
+    Normalizer that combines system messages with the following user message using generic instruction tags.
     """
 
     async def normalize_async(self, messages: list[Message]) -> list[Message]:
         """
-        Return messages with the first system message combined into the first user message.
+        Return messages with each system message combined into the following user message.
 
         The format uses generic instruction tags:
         ### Instructions ###
@@ -25,43 +25,81 @@ async def normalize_async(self, messages: list[Message]) -> list[Message]:
             messages: The list of messages to normalize.
 
         Returns:
-            A Message with the system message squashed into the first user message.
+            Messages with system instructions squashed into the following user message.
 
         Raises:
             ValueError: If the messages list is empty.
         """
         if not messages:
             raise ValueError("Messages list cannot be empty")
 
-        # Check if first message is a system message
-        first_piece = messages[0].get_piece()
-        if first_piece.api_role != "system":
-            # No system message to squash, return messages unchanged
+        system_messages = [message for message in messages if message.api_role == "system"]
+        if not system_messages:
             return list(messages)
 
-        if len(messages) == 1:
-            # Only system message, convert to user message.
-            return [
-                build_squashed_user_message(
-                    new_message_content=first_piece.converted_value, source_messages=messages[:1]
-                )
-            ]
+        result: list[Message] = []
+        index = 0
+        while index < len(messages):
+            message = messages[index]
+            if message.api_role != "system":
+                result.append(message)
+                index += 1
+                continue
 
-        user_message_index = next(
-            (i for i, message in enumerate(messages[1:], start=1) if message.api_role == "user"),
-            -1,
-        )
-        if user_message_index == -1:
-            # Preserve the instruction content without rewriting non-user messages.
-            return [
-                build_squashed_user_message(
-                    new_message_content=first_piece.converted_value, source_messages=messages[:1]
+            system_messages = [message]
+            index += 1
+            while index < len(messages) and messages[index].api_role == "system":
+                system_messages.append(messages[index])
+                index += 1
+
+            if index < len(messages) and messages[index].api_role == "user":
+                result.append(
+                    self._squash_system_messages_into_user(
+                        system_messages=system_messages,
+                        user_message=messages[index],
+                    )
+                )
+                index += 1
+            else:
+                result.append(
+                    build_squashed_user_message(
+                        new_message_content=self._get_system_content(system_messages),
+                        source_messages=system_messages,
+                    )
                 )
-            ] + list(messages[1:])
 
-        # Combine system with the first user message, preserving non-text pieces (e.g. images) and their order.
-        system_content = first_piece.converted_value
-        user_message = messages[user_message_index]
+        return result
+
+    @staticmethod
+    def _get_system_content(system_messages: list[Message]) -> str:
+        """
+        Combine system-message pieces in message order.
+
+        Args:
+            system_messages: The system messages to combine.
+
+        Returns:
+            The combined system-message content.
+        """
+        return "\n\n".join(piece.converted_value for message in system_messages for piece in message.message_pieces)
+
+    def _squash_system_messages_into_user(
+        self,
+        *,
+        system_messages: list[Message],
+        user_message: Message,
+    ) -> Message:
+        """
+        Merge system instructions into a user message while preserving its pieces.
+
+        Args:
+            system_messages: The system messages to merge.
+            user_message: The following user message.
+
+        Returns:
+            The user message with the system instructions applied.
+        """
+        system_content = self._get_system_content(system_messages)
         # Propagate prompt_metadata from the user message's first piece so downstream normalizers
         # (e.g. JsonSchemaNormalizer) still see request-level metadata after squashing.
         propagated_metadata = dict(user_message.message_pieces[0].prompt_metadata)
@@ -96,7 +134,4 @@ async def normalize_async(self, messages: list[Message]) -> list[Message]:
                 + list(user_message.message_pieces[text_piece_index + 1 :])
             )
 
-        squashed_message = Message(message_pieces=squashed_pieces)
-
-        # Remove system (index 0), replace the first user message with the squashed version, preserve all others
-        return list(messages[1:user_message_index]) + [squashed_message] + list(messages[user_message_index + 1 :])
+        return Message(message_pieces=squashed_pieces)

pyrit/message_normalizer/generic_system_squash.py

@@ -13,7 +13,7 @@
 """
 How to handle system messages in models with varying support:
 - "keep": Keep system messages as-is (default for most models)
-- "squash": Merge system message into first user message
+- "squash": Merge system messages into the following user message
 - "ignore": Drop system messages entirely
 """
 
@@ -90,7 +90,7 @@ async def apply_system_message_behavior_async(
         messages: The list of Message objects to process.
         behavior: How to handle system messages:
             - "keep": Return messages unchanged
-            - "squash": Merge system into first user message
+            - "squash": Merge system messages into the following user message
             - "ignore": Remove system messages
 
     Returns: