Add multi-agent capabilties

README.md

@@ -25,6 +25,7 @@ npm install @agentesting/agentest --save-dev
 | | |
 |---|---|
 | Scenario-based tests | Define user personas, goals, and knowledge — Agentest generates realistic multi-turn conversations |
+| Scripted multi-turn | Predetermined user messages with per-turn trajectory assertions for deterministic regression tests |
 | Tool-call mocks | Intercept and control tool calls with functions, sequences, and error simulation |
 | Trajectory assertions | Verify tool call order and arguments with `strict`, `contains`, `unordered`, and `within` match modes |
 | LLM-as-judge metrics | Helpfulness, coherence, relevance, faithfulness, goal completion, behavior failure detection |
@@ -205,6 +206,44 @@ scenario('user books a morning slot', {
 })
 ```
 
+#### Scripted alternative
+
+For deterministic tests with predetermined user messages and per-turn assertions:
+
+```ts
+// tests/context.sim.ts
+import { scenario } from '@agentesting/agentest'
+
+scenario('follow-up reuses vehicle context', {
+  turns: [
+    {
+      userMessage: 'How fast was Leo (12345678) last week?',
+      assertions: {
+        toolCalls: {
+          matchMode: 'contains',
+          expected: [{ name: 'get_speed', args: { id: '12345678' }, argMatchMode: 'partial' }],
+        },
+      },
+    },
+    {
+      userMessage: 'And what about its failure count?',
+      assertions: {
+        toolCalls: {
+          matchMode: 'contains',
+          expected: [{ name: 'get_failures', args: { id: '12345678' }, argMatchMode: 'partial' }],
+        },
+      },
+    },
+  ],
+  mocks: {
+    tools: {
+      get_speed: () => ({ speed: 0.8, unit: 'm/s' }),
+      get_failures: () => ({ count: 5 }),
+    },
+  },
+})
+```
+
 ### 3. Run
 
 ```bash
@@ -337,11 +376,27 @@ export default defineConfig({
 })
 ```
 
-The handler receives the full message history (same `ChatMessage` format used internally) and must return an assistant message. If the response includes `tool_calls`, Agentest runs them through mocks and calls your handler again with the tool results — the same loop as with HTTP endpoints.
+The handler receives the full message history and a `ctx` object. If the response includes `tool_calls`, Agentest runs them through mocks and calls your handler again with the tool results — the same loop as with HTTP endpoints.
+
+The `ctx` object provides `resolveTool()` for agents that handle tools internally (e.g., multi-agent supervisors):
+
+```ts
+handler: async (messages, ctx) => {
+  const mockClient = {
+    async get(endpoint, params) {
+      return ctx.resolveTool(endpoint, params)  // uses scenario mocks, records for trajectory
+    },
+  }
+  const agent = createSupervisor({ client: mockClient })
+  const result = await agent.invoke({ messages })
+  return { role: 'assistant' as const, content: result.content }
+}
+```
 
 This is useful when your agent:
 - Uses a non-OpenAI API (Anthropic, Google, custom protocols)
 - Runs in-process (no HTTP server needed)
+- Has multi-agent routing that handles tools internally
 - Needs custom request/response mapping
 - Uses an SDK or framework with its own calling convention
 

docs/examples/multi-turn.md

@@ -2,7 +2,11 @@
 
 Test complex scenarios that require multiple interactions.
 
-## Example: Multi-step Booking Flow
+## Simulated Multi-turn (LLM-driven)
+
+Use `profile` and `goal` to let the simulated user drive the conversation autonomously. Agentest's LLM-powered user generates realistic messages and decides when the goal is met.
+
+### Example: Multi-step Booking Flow
 
 ```ts
 import { scenario, sequence } from '@agentesting/agentest'
@@ -61,7 +65,7 @@ scenario('user completes multi-step booking with questions', {
 })
 ```
 
-## Expected Conversation Flow
+### Expected Conversation Flow
 
 ```
 Turn 1:
@@ -89,7 +93,7 @@ Agent: *calls create_booking*
 Agent: Perfect! Booked for Tuesday at 09:00. Confirmation: BK-001
 ```
 
-## Testing Conversation Depth
+### Testing Conversation Depth
 
 Use `maxTurns` to accommodate longer conversations:
 
@@ -101,7 +105,7 @@ scenario('complex troubleshooting scenario', {
 })
 ```
 
-## Multiple Conversations
+### Multiple Conversations
 
 Run more conversations to test variance:
 
@@ -112,3 +116,181 @@ scenario('unpredictable user behavior', {
   conversationsPerScenario: 10,  // Run 10 times
 })
 ```
+
+## Scripted Multi-turn (Deterministic)
+
+Use `turns` to define exact user messages for each turn. This skips the simulated user entirely — no LLM is used to generate messages. Each turn can have its own trajectory assertions.
+
+This is ideal for testing **context carry-forward**, **conversation continuity**, and **deterministic regression tests**.
+
+### Example: Context Carry-forward
+
+```ts
+import { scenario } from '@agentesting/agentest'
+
+scenario('follow-up reuses vehicle context', {
+  turns: [
+    {
+      userMessage: 'How fast was Leo (12345678) last week?',
+      assertions: {
+        toolCalls: {
+          matchMode: 'contains',
+          expected: [
+            { name: 'performance_agent', args: { serials: '12345678' }, argMatchMode: 'partial' },
+          ],
+        },
+      },
+    },
+    {
+      userMessage: 'And what about its failure count in the same period?',
+      assertions: {
+        toolCalls: {
+          matchMode: 'contains',
+          expected: [
+            { name: 'failure_agent', args: { serials: '12345678' }, argMatchMode: 'partial' },
+          ],
+        },
+      },
+    },
+  ],
+
+  mocks: {
+    tools: {
+      performance_agent: () => ({ speed: 0.8, unit: 'm/s' }),
+      failure_agent: () => ({ count: 5, severity_breakdown: { warning: 3, error: 2 } }),
+    },
+  },
+})
+```
+
+The second turn says "its failure count" — the agent must carry forward that "it" refers to vehicle `12345678` from the previous turn. The per-turn assertion verifies this.
+
+### Example: Domain Switch with Follow-up Export
+
+```ts
+scenario('cross-domain pivot then export', {
+  turns: [
+    {
+      userMessage: 'What was the energy consumption of Leo (12345678) from Jan 1 to Jan 7?',
+      assertions: {
+        toolCalls: {
+          matchMode: 'contains',
+          expected: [{ name: 'performance_agent', argMatchMode: 'ignore' }],
+        },
+      },
+    },
+    {
+      userMessage: 'Were there any errors during that period?',
+      assertions: {
+        toolCalls: {
+          matchMode: 'contains',
+          expected: [
+            { name: 'failure_agent', args: { serials: '12345678' }, argMatchMode: 'partial' },
+          ],
+        },
+      },
+    },
+    {
+      userMessage: 'Export that to CSV',
+      assertions: {
+        toolCalls: {
+          matchMode: 'contains',
+          expected: [{ name: 'export_to_csv', argMatchMode: 'ignore' }],
+        },
+      },
+    },
+  ],
+
+  mocks: {
+    tools: {
+      performance_agent: () => ({ energy: 12.4, unit: 'kWh' }),
+      failure_agent: () => ({ count: 3 }),
+      export_to_csv: () => ({ fileId: 'export-001', url: '/download/export-001' }),
+    },
+  },
+})
+```
+
+### Key Differences from Simulated Multi-turn
+
+| | Simulated (`profile` + `goal`) | Scripted (`turns`) |
+|---|---|---|
+| User messages | Generated by LLM | Predetermined |
+| Deterministic | No (LLM variance) | Yes |
+| `conversationsPerScenario` default | From config (usually 3) | 1 |
+| LLM evaluation | Full metrics + goal completion | Only if `goal` is provided |
+| Per-turn assertions | No (cumulative only) | Yes |
+| Best for | Exploratory testing, persona variance | Regression tests, context carry-forward |
+
+### Scripted Scenarios with Evaluation
+
+To enable LLM-as-judge evaluation on scripted scenarios, provide a `goal`:
+
+```ts
+scenario('follow-up with quality evaluation', {
+  goal: 'Get speed and failure data for vehicle Leo.',
+
+  turns: [
+    { userMessage: 'How fast was Leo (12345678) last week?' },
+    { userMessage: 'And what about its failure count?' },
+  ],
+
+  mocks: {
+    tools: {
+      performance_agent: () => ({ speed: 0.8 }),
+      failure_agent: () => ({ count: 5 }),
+    },
+  },
+})
+```
+
+Without a `goal`, scripted scenarios skip LLM evaluation entirely and rely on trajectory assertions for pass/fail.
+
+### Cumulative + Per-turn Assertions
+
+You can combine scenario-level cumulative assertions with per-turn assertions:
+
+```ts
+scenario('full booking flow', {
+  // Cumulative: these tools must all be called across the whole conversation
+  assertions: {
+    toolCalls: {
+      matchMode: 'contains',
+      expected: [
+        { name: 'check_availability', argMatchMode: 'ignore' },
+        { name: 'create_booking', argMatchMode: 'ignore' },
+      ],
+    },
+  },
+
+  turns: [
+    {
+      userMessage: 'Is Tuesday morning available?',
+      // Per-turn: only check_availability should be called in this turn
+      assertions: {
+        toolCalls: {
+          matchMode: 'contains',
+          expected: [{ name: 'check_availability', argMatchMode: 'ignore' }],
+        },
+      },
+    },
+    {
+      userMessage: 'Book the 9am slot.',
+      // Per-turn: create_booking should be called in this turn
+      assertions: {
+        toolCalls: {
+          matchMode: 'contains',
+          expected: [{ name: 'create_booking', argMatchMode: 'ignore' }],
+        },
+      },
+    },
+  ],
+
+  mocks: {
+    tools: {
+      check_availability: () => ({ available: true, slots: ['09:00', '10:30'] }),
+      create_booking: () => ({ success: true, bookingId: 'BK-001' }),
+    },
+  },
+})
+```

docs/guide/pass-fail-logic.md

@@ -11,7 +11,7 @@ A scenario either passes or fails based on multiple criteria. The overall run pa
 A scenario **passes** when all of the following conditions are true:
 
 1. ✅ No conversation threw an error
-2. ✅ All trajectory assertions matched (if configured)
+2. ✅ All trajectory assertions matched (if configured) — both scenario-level cumulative and per-turn
 3. ✅ No errors at or above the configured `failOnErrorSeverity`
 4. ✅ All metric averages meet their configured thresholds