diff --git a/.changeset/tool-output-sandbox-overflow.md b/.changeset/tool-output-sandbox-overflow.md new file mode 100644 index 0000000000..6679791d97 --- /dev/null +++ b/.changeset/tool-output-sandbox-overflow.md @@ -0,0 +1,5 @@ +--- +"eve": patch +--- + +Agents can now opt into a model-facing tool output limit that writes oversized results to the session sandbox while preserving the full `action.result` for hooks, channels, clients, and observability. Successful spills emit a durable `tool.output.spilled` event and bounded trace metadata. diff --git a/docs/agent-config.md b/docs/agent-config.md index cb9c4d7ae7..b59bb83d3a 100644 --- a/docs/agent-config.md +++ b/docs/agent-config.md @@ -1,6 +1,6 @@ --- title: "Agents" -description: "Configure an eve agent's model, reasoning effort, compaction, limits, and runtime behavior in agent.ts." +description: "Configure an eve agent's model, reasoning effort, compaction, tool output, limits, and runtime behavior in agent.ts." --- An eve app has one root agent assembled from the files under `agent/`. Its optional `agent.ts` calls `defineAgent` (from `eve`) when you need to configure the model or other runtime behavior. Declared [subagents](./subagents) have their own `agent.ts` and capabilities; this page covers the configuration shared by root agents and subagents. @@ -147,6 +147,53 @@ export default defineAgent({ See [Default harness](./concepts/default-harness#compaction) for how the loop applies it. +## Spill oversized tool results + +Set `toolOutput` to keep one tool result from filling model history. The policy +applies across authored tools, framework tools, generated connection tools, +client-supplied tools, and subagent results that eve sends to the model: + +```ts title="agent/agent.ts" +export default defineAgent({ + model: "anthropic/claude-opus-4.8", + toolOutput: { + maxInlineBytes: 64 * 1024, + overflow: "sandbox", + }, +}); +``` + +Both fields are required. `maxInlineBytes` must be a positive integer. The +policy is disabled when `toolOutput` is omitted. eve measures text as UTF-8 +bytes and JSON by its compact serialized value. A result at or below +`maxInlineBytes` stays unchanged. A larger result is written under +`/workspace/.eve/tool-results` in the session sandbox: text as `.txt`, JSON as +readable `.json`. Model history receives a small object with the file path, +original byte count, and tool name. Replaying the same call with the same +result overwrites the same path; a changed result receives a different path. + +The policy runs after a tool's `toModelOutput`, so use that callback when you +can give the model a smaller semantic summary. The full execution result still +appears in `action.result` for hooks, channels, clients, and observability; +only model history receives the file reference. Framework control results that +eve must reread to reconstruct capabilities, such as `connection_search`, and +approval denials remain inline. + +After the sandbox write succeeds, eve emits a durable `tool.output.spilled` +stream event with the call id, tool name, original byte count, configured +limit, file path, and deterministic `spillId`. Hooks, channels, and raw stream +clients can observe the spill without inspecting or changing `action.result`. +OpenTelemetry instrumentation records the same bounded metadata on an +`agent.tool.output` span. Durable retries may physically deliver the event +more than once; deduplicate notifications by `spillId`, not the stream event +id. + +Provider-executed server tools are outside this boundary: the provider consumes +their output before eve receives it, so `toolOutput` cannot limit that first +provider call. Sandbox references follow the [sandbox lifecycle](./sandbox#lifecycle); +persist important artifacts outside the sandbox when they must survive +provider-side sandbox replacement. + ## Runtime limits Use `limits` for framework-owned runtime caps. Session token limits stop the @@ -255,6 +302,7 @@ installed package must stay external in hosted output, list it in | `limits` | `AgentLimitsDefinition` | field-specific | Framework-owned runtime limits. Sessions complete after 30 days by default; token-limit defaults and inheritance are described above. Set a limit to `false` to disable it. | | `experimental` | `{ workflow?: { world?: string } }` | unset | Opt-in settings that can change or disappear in any release. Treat them as unstable. `workflow.world` selects the Workflow world package backing session state, queues, hooks, and streams on the root agent. | | `outputSchema` | Standard Schema or a JSON Schema object | none | Structured return type for function-like invocations such as a subagent turn, schedule, or remote job. Ordinary interactive turns ignore it unless the client supplies a per-message schema. | +| `toolOutput` | `AgentToolOutputDefinition` | none | Optional agent-wide overflow policy for model-facing tool results. Oversized text and JSON are written to the session sandbox while `action.result` keeps the full output. | | `build` | `{ externalDependencies?: string[] }` | none | Hosted-build packaging controls. `externalDependencies` keeps listed packages external while eve compiles authored modules such as tools and channels, and traces those packages into the hosted output. | `externalDependencies` is a packaging control only. It keeps selected packages as runtime dependencies in the hosted output; it does not authorize, configure, or review any third-party service those packages may call. diff --git a/docs/concepts/sessions-runs-and-streaming.md b/docs/concepts/sessions-runs-and-streaming.md index 0249b64579..de378cbfb8 100644 --- a/docs/concepts/sessions-runs-and-streaming.md +++ b/docs/concepts/sessions-runs-and-streaming.md @@ -53,6 +53,7 @@ The stream is newline-delimited JSON (NDJSON), one event per line: | `actions.requested` | The model requested one or more actions, including tool calls; calls stream before execution. | | `action.partial` | A locally executed tool generator yielded a preliminary output snapshot. | | `action.result` | A tool call returned. | +| `tool.output.spilled` | An oversized model-facing tool result moved to the sandbox; carries bounded reference metadata. | | `input.requested` | The run paused for human input ([HITL](/docs/human-in-the-loop) approval or `ask_question`); carries `requests`. | | `input.resolved` | The server accepted terminal human-input outcomes; carries `resolutions` with responses when provided. | | `subagent.called` | A subagent was delegated; carries `childSessionId` to attach to. | @@ -83,6 +84,11 @@ When a streamed tool input becomes a validated call, its `action.input.appended` `action.partial` carries one complete preliminary output snapshot from an authored async-generator tool. A later partial for the same `callId` replaces it, and `action.result` is the final snapshot. When the durable writer is busy, eve may keep only the newest adjacent partial for a call. Treat partials as last-write-wins: a durable step can retry and replay overlapping event runs. Provider-executed tool progress and MCP progress notifications are not projected as `action.partial` events. +`tool.output.spilled` leaves the complete `action.result` unchanged. Its data +contains `callId`, `toolName`, `bytes`, `maxInlineBytes`, `path`, and a +deterministic `spillId`. Durable step retries can repeat the notification with +a different `meta.id`; deduplicate spill handling by `spillId`. + Note: consider the privacy, confidentiality, and user-experience implications for displaying, storing, or transmitting reasoning events in your application. `message.completed` can fire more than once in a turn: the agent often emits interim assistant text before a tool call. To tell tool-call narration from a terminal reply, check `message.completed.data.finishReason`. `step.completed.data.finishReason` mirrors the step outcome, and usage lives on `step.completed`. diff --git a/docs/guides/client/streaming.mdx b/docs/guides/client/streaming.mdx index edcd8623e1..cc9553f55b 100644 --- a/docs/guides/client/streaming.mdx +++ b/docs/guides/client/streaming.mdx @@ -98,21 +98,22 @@ function handleEvent(event: MessageStreamEvent) { The most common UI events are: -| Event | Use | -| ----------------------- | ------------------------------------------------------------------------------ | -| `message.received` | Confirm the user message landed; `data.parts` includes text and file metadata. | -| `reasoning.appended` | Render reasoning deltas when the model provides them. | -| `message.appended` | Render assistant text deltas. | -| `action.input.appended` | Accumulate raw tool-input deltas before validation completes. | -| `actions.requested` | Show tool calls as the model requests them, before execution. | -| `action.partial` | Update a generator tool's provisional output snapshot. | -| `action.result` | Show tool call results. | -| `input.requested` | Pause the UI for approval or a question answer. | -| `input.resolved` | Record the server-accepted outcome and response for each human-input request. | -| `result.completed` | Read structured output from an [output schema](./output-schema). | -| `session.waiting` | Enable the composer; the same fixed session handle accepts the next message. | -| `session.completed` | Mark the conversation terminal. | -| `session.failed` | Mark the conversation failed. | +| Event | Use | +| ----------------------- | ------------------------------------------------------------------------------- | +| `message.received` | Confirm the user message landed; `data.parts` includes text and file metadata. | +| `reasoning.appended` | Render reasoning deltas when the model provides them. | +| `message.appended` | Render assistant text deltas. | +| `action.input.appended` | Accumulate raw tool-input deltas before validation completes. | +| `actions.requested` | Show tool calls as the model requests them, before execution. | +| `action.partial` | Update a generator tool's provisional output snapshot. | +| `action.result` | Show tool call results. | +| `tool.output.spilled` | Observe that the model received a sandbox reference instead of the full result. | +| `input.requested` | Pause the UI for approval or a question answer. | +| `input.resolved` | Record the server-accepted outcome and response for each human-input request. | +| `result.completed` | Read structured output from an [output schema](./output-schema). | +| `session.waiting` | Enable the composer; the same fixed session handle accepts the next message. | +| `session.completed` | Mark the conversation terminal. | +| `session.failed` | Mark the conversation failed. | For the complete event table, see [Sessions, runs & streaming](../../concepts/sessions-runs-and-streaming). diff --git a/docs/tools/overview.mdx b/docs/tools/overview.mdx index 7513490a0b..f96a9b2bff 100644 --- a/docs/tools/overview.mdx +++ b/docs/tools/overview.mdx @@ -142,6 +142,15 @@ toModelOutput(output) { Tool outputs must be JSON-serializable. Return plain objects, arrays, strings, numbers, booleans, or `null`; convert values like `Date`, `Map`, `Set`, `NaN`, and cyclic objects before returning them from `execute` or from a `{ type: "json" }` `toModelOutput`. +For an agent-wide safety bound, set `toolOutput` in +[`defineAgent`](/docs/agent-config#spill-oversized-tool-results). It applies +after `toModelOutput` and also covers generated connection tools that do not +have an authored projection. Oversized model-facing output moves to the +session sandbox while `action.result` keeps the full execution result. +Successful spills also emit a durable `tool.output.spilled` stream event, so +hooks, channels, clients, and traces can observe the model-facing projection +without changing the complete `action.result`. + ### Send images to the model with content parts A tool that produces an image — a screenshot, a rendered chart — can hand the pixels to a vision-capable model by returning a `content` output from `toModelOutput`. Build outputs with the `toolOutput` helpers and parts with the `toolOutputPart` helpers, both from `eve/tools`: diff --git a/e2e/fixtures/agent-tools-sandbox/agent/agent.ts b/e2e/fixtures/agent-tools-sandbox/agent/agent.ts index 95fd9acdd0..7506410cf7 100644 --- a/e2e/fixtures/agent-tools-sandbox/agent/agent.ts +++ b/e2e/fixtures/agent-tools-sandbox/agent/agent.ts @@ -5,4 +5,8 @@ import { respond } from "./lib/mock-responder.js"; export default defineAgent({ ...e2eAgentConfig({ mock: respond }), reasoning: "high", + toolOutput: { + maxInlineBytes: 64 * 1024, + overflow: "sandbox", + }, }); diff --git a/e2e/fixtures/agent-tools-sandbox/agent/lib/mock-responder.ts b/e2e/fixtures/agent-tools-sandbox/agent/lib/mock-responder.ts index e3676fbe19..fae1f5d731 100644 --- a/e2e/fixtures/agent-tools-sandbox/agent/lib/mock-responder.ts +++ b/e2e/fixtures/agent-tools-sandbox/agent/lib/mock-responder.ts @@ -1,9 +1,12 @@ import type { MockModelRequest, MockModelResponse } from "eve/evals"; +import { OVERFLOW_PROBE_TOKEN } from "./overflow-probe.js"; const SUBAGENT_DIRECTIVE = /ask the `([^`]+)` subagent with message:\s*([\s\S]+)/iu; const BASH_DIRECTIVE = /run the bash command `([^`]+)`/iu; const SKILL_DIRECTIVE = /load the `([^`]+)` skill/iu; +const OVERFLOW_DIRECTIVE = /run the `overflow_probe` tool/iu; +const OVERFLOW_FILE_PATH = /^\/workspace\/\.eve\/tool-results\/[a-f0-9]{64}\.json$/u; /** * Scripted mock for the world suites: sandbox evals phrase every prompt as an * explicit directive, so the responder executes exactly the requested tool @@ -22,6 +25,9 @@ export function respond(request: MockModelRequest): MockModelResponse | string { } } const turnHasToolResult = lastToolResultIndex > lastAuthoredUserIndex; + if (OVERFLOW_DIRECTIVE.test(message)) { + return overflowProbeResponse(request); + } const subagent = SUBAGENT_DIRECTIVE.exec(message); if (subagent?.[1] !== undefined && subagent[2] !== undefined) { @@ -55,6 +61,54 @@ export function respond(request: MockModelRequest): MockModelResponse | string { return `Mock reply: ${message}`; } +function overflowProbeResponse(request: MockModelRequest): MockModelResponse | string { + const probe = request.toolResults.find((result) => result.id === "overflow-probe-source"); + if (probe === undefined) { + return { + toolCalls: [{ id: "overflow-probe-source", input: {}, name: "overflow_probe" }], + }; + } + + if (request.toolResults.some((result) => result.id === "overflow-probe-read")) { + return bashStdout(request); + } + + const path = overflowFilePath(probe.output); + if (path === undefined) { + return "overflow_probe did not produce an eve tool-output file reference"; + } + + return { + toolCalls: [ + { + id: "overflow-probe-read", + input: { command: `grep -m 1 -o '${OVERFLOW_PROBE_TOKEN}' ${path}` }, + name: "bash", + }, + ], + }; +} + +function overflowFilePath(output: unknown): string | undefined { + if ( + typeof output !== "object" || + output === null || + !("kind" in output) || + output.kind !== "eve-tool-output-file" || + !("bytes" in output) || + typeof output.bytes !== "number" || + output.bytes <= 64 * 1024 || + !("toolName" in output) || + output.toolName !== "overflow_probe" || + !("path" in output) || + typeof output.path !== "string" || + !OVERFLOW_FILE_PATH.test(output.path) + ) { + return undefined; + } + return output.path; +} + function bashStdout(request: MockModelRequest): string { const output = [...request.toolResults] .reverse() diff --git a/e2e/fixtures/agent-tools-sandbox/agent/lib/overflow-probe.ts b/e2e/fixtures/agent-tools-sandbox/agent/lib/overflow-probe.ts new file mode 100644 index 0000000000..4365f72315 --- /dev/null +++ b/e2e/fixtures/agent-tools-sandbox/agent/lib/overflow-probe.ts @@ -0,0 +1,2 @@ +export const OVERFLOW_PROBE_PAYLOAD_BYTES = 128 * 1024; +export const OVERFLOW_PROBE_TOKEN = "tool-output-overflow-ok-R4V"; diff --git a/e2e/fixtures/agent-tools-sandbox/agent/tools/overflow_probe.ts b/e2e/fixtures/agent-tools-sandbox/agent/tools/overflow_probe.ts new file mode 100644 index 0000000000..fe35f562ad --- /dev/null +++ b/e2e/fixtures/agent-tools-sandbox/agent/tools/overflow_probe.ts @@ -0,0 +1,15 @@ +import { defineTool } from "eve/tools"; +import { z } from "zod"; +import { OVERFLOW_PROBE_PAYLOAD_BYTES, OVERFLOW_PROBE_TOKEN } from "../lib/overflow-probe.js"; + +export default defineTool({ + description: + "Smoke-test fixture: returns deterministic oversized JSON. Only call when explicitly asked to use `overflow_probe`.", + inputSchema: z.object({}), + execute() { + return { + marker: OVERFLOW_PROBE_TOKEN, + payload: "x".repeat(OVERFLOW_PROBE_PAYLOAD_BYTES), + }; + }, +}); diff --git a/e2e/fixtures/agent-tools-sandbox/evals/sandbox/tool-output-overflow.eval.ts b/e2e/fixtures/agent-tools-sandbox/evals/sandbox/tool-output-overflow.eval.ts new file mode 100644 index 0000000000..f0665a6aa5 --- /dev/null +++ b/e2e/fixtures/agent-tools-sandbox/evals/sandbox/tool-output-overflow.eval.ts @@ -0,0 +1,39 @@ +import { defineEval } from "eve/evals"; + +const OVERFLOW_PROBE_PAYLOAD_BYTES = 128 * 1024; +const OVERFLOW_PROBE_TOKEN = "tool-output-overflow-ok-R4V"; + +export default defineEval({ + description: + "Tool output overflow: oversized model history becomes a readable sandbox reference while action.result stays complete.", + async test(t) { + await t.send( + [ + "Run the `overflow_probe` tool exactly once.", + `Read the generated eve tool-output file by running the bash command \`grep -m 1 -o '${OVERFLOW_PROBE_TOKEN}' \` with the actual reference path.`, + `Reply with exactly ${OVERFLOW_PROBE_TOKEN}.`, + ].join("\n"), + ); + + t.succeeded(); + t.calledTool("overflow_probe", { + count: 1, + output: hasCompleteProbeOutput, + }); + t.calledTool("bash", { + count: 1, + output: new RegExp(OVERFLOW_PROBE_TOKEN), + }); + t.messageIncludes(OVERFLOW_PROBE_TOKEN); + }, +}); + +function hasCompleteProbeOutput(value: unknown): boolean { + if (typeof value !== "object" || value === null) return false; + const output = value as { marker?: unknown; payload?: unknown }; + return ( + output.marker === OVERFLOW_PROBE_TOKEN && + typeof output.payload === "string" && + output.payload.length === OVERFLOW_PROBE_PAYLOAD_BYTES + ); +} diff --git a/packages/eve/extension-contracts/compatibility/channel/v10.ts b/packages/eve/extension-contracts/compatibility/channel/v10.ts new file mode 100644 index 0000000000..67e58f0bbe --- /dev/null +++ b/packages/eve/extension-contracts/compatibility/channel/v10.ts @@ -0,0 +1,3 @@ +import { disableRoute } from "#public/channels/index.js"; + +export default disableRoute(); diff --git a/packages/eve/extension-contracts/compatibility/dynamicInstructions/v14.ts b/packages/eve/extension-contracts/compatibility/dynamicInstructions/v14.ts new file mode 100644 index 0000000000..02a61cf705 --- /dev/null +++ b/packages/eve/extension-contracts/compatibility/dynamicInstructions/v14.ts @@ -0,0 +1,10 @@ +import { defineDynamic, defineInstructions } from "#public/instructions/index.js"; + +export default defineDynamic({ + events: { + "session.started": (_event, ctx) => + defineInstructions({ + markdown: `Use session ${ctx.session.id} when correlating evidence.`, + }), + }, +}); diff --git a/packages/eve/extension-contracts/compatibility/dynamicSkill/v13.ts b/packages/eve/extension-contracts/compatibility/dynamicSkill/v13.ts new file mode 100644 index 0000000000..3c6e1498e9 --- /dev/null +++ b/packages/eve/extension-contracts/compatibility/dynamicSkill/v13.ts @@ -0,0 +1,11 @@ +import { defineDynamic, defineSkill } from "#public/skills/index.js"; + +export default defineDynamic({ + events: { + "turn.started": (_event, ctx) => + defineSkill({ + description: `Review evidence for session ${ctx.session.id}.`, + markdown: "# Evidence review\n\nCheck every claim against its source.", + }), + }, +}); diff --git a/packages/eve/extension-contracts/compatibility/dynamicTool/v20.ts b/packages/eve/extension-contracts/compatibility/dynamicTool/v20.ts new file mode 100644 index 0000000000..6252c112c4 --- /dev/null +++ b/packages/eve/extension-contracts/compatibility/dynamicTool/v20.ts @@ -0,0 +1,19 @@ +import { z } from "zod"; + +import { defineDynamic, defineTool } from "#public/tools/index.js"; + +export default defineDynamic({ + events: { + "turn.started": (_event, ctx) => + defineTool({ + description: "Echo the tool call identifiers.", + inputSchema: z.object({ note: z.string() }), + execute: ({ note }, toolCtx) => ({ + callId: toolCtx.callId, + note, + sessionId: ctx.session.id, + toolName: toolCtx.toolName, + }), + }), + }, +}); diff --git a/packages/eve/extension-contracts/compatibility/hook/v15.ts b/packages/eve/extension-contracts/compatibility/hook/v15.ts new file mode 100644 index 0000000000..38ecc7e065 --- /dev/null +++ b/packages/eve/extension-contracts/compatibility/hook/v15.ts @@ -0,0 +1,13 @@ +import { defineHook } from "#public/hooks/index.js"; + +export default defineHook({ + events: { + "subagent.completed"(event, ctx) { + console.info("subagent completed", { + output: event.data.output, + sessionId: ctx.session.id, + subagentName: event.data.subagentName, + }); + }, + }, +}); diff --git a/packages/eve/extension-contracts/compatibility/schedule/v4.ts b/packages/eve/extension-contracts/compatibility/schedule/v4.ts new file mode 100644 index 0000000000..15ba030ed8 --- /dev/null +++ b/packages/eve/extension-contracts/compatibility/schedule/v4.ts @@ -0,0 +1,18 @@ +import channel from "../channel/v7.js"; +import { defineSchedule } from "#public/schedules/index.js"; + +export default defineSchedule({ + cron: "0 0 * * *", + async run({ appAuth, to, waitUntil }) { + waitUntil( + (async () => { + const session = await to(channel, { sessionRef: "daily" }).send("Start review", { + auth: appAuth, + }); + await session.respond([{ optionId: "approve", requestId: "approval-1" }], { + auth: appAuth, + }); + })(), + ); + }, +}); diff --git a/packages/eve/extension-contracts/compatibility/subagent/v5.ts b/packages/eve/extension-contracts/compatibility/subagent/v5.ts new file mode 100644 index 0000000000..e3bdcaa242 --- /dev/null +++ b/packages/eve/extension-contracts/compatibility/subagent/v5.ts @@ -0,0 +1,7 @@ +import { defineAgent } from "#public/index.js"; + +export default defineAgent({ + compaction: { thresholdPercent: 0.8 }, + description: "Delegate research tasks.", + model: "anthropic/claude-sonnet-5", +}); diff --git a/packages/eve/extension-contracts/entrypoints/subagent.ts b/packages/eve/extension-contracts/entrypoints/subagent.ts index 91cd73327b..d0b258f528 100644 --- a/packages/eve/extension-contracts/entrypoints/subagent.ts +++ b/packages/eve/extension-contracts/entrypoints/subagent.ts @@ -3,6 +3,7 @@ export { type AgentDefinition, type AgentModelDefinition, type AgentStaticModelDefinition, + type AgentToolOutputDefinition, type DefinedAgent, type DynamicLocalSubagentDefinition, type DynamicSubagentDefinition, diff --git a/packages/eve/extension-contracts/reports/channel/v11.json b/packages/eve/extension-contracts/reports/channel/v11.json new file mode 100644 index 0000000000..47e708fbb8 --- /dev/null +++ b/packages/eve/extension-contracts/reports/channel/v11.json @@ -0,0 +1,21 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "channel", + "epoch": 11, + "sha256": "84e7672b83347990c9ceaf5eeef4e75b2b95e0ec6942622cf98b106b0c0f1afb", + "exports": [ + "DELETE", + "GET", + "HEAD", + "OPTIONS", + "PATCH", + "POST", + "PUT", + "WS", + "createWebSocketUpgradeServer", + "defineChannel", + "disableRoute", + "isChannel", + "isDisabledRouteSentinel" + ] +} diff --git a/packages/eve/extension-contracts/reports/dynamicInstructions/v15.json b/packages/eve/extension-contracts/reports/dynamicInstructions/v15.json new file mode 100644 index 0000000000..6221c946ad --- /dev/null +++ b/packages/eve/extension-contracts/reports/dynamicInstructions/v15.json @@ -0,0 +1,7 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "dynamicInstructions", + "epoch": 15, + "sha256": "09f69371fa940383327d5c8a72284a9932136a0d33da9af3a5c1b78796cf5c04", + "exports": ["defineDynamic"] +} diff --git a/packages/eve/extension-contracts/reports/dynamicSkill/v14.json b/packages/eve/extension-contracts/reports/dynamicSkill/v14.json new file mode 100644 index 0000000000..c654ad685f --- /dev/null +++ b/packages/eve/extension-contracts/reports/dynamicSkill/v14.json @@ -0,0 +1,7 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "dynamicSkill", + "epoch": 14, + "sha256": "09a178f434140bd491fe39c1f5dfd4a23e7eb22699ccb0a3b399125d7dca569d", + "exports": ["defineDynamic"] +} diff --git a/packages/eve/extension-contracts/reports/dynamicTool/v21.json b/packages/eve/extension-contracts/reports/dynamicTool/v21.json new file mode 100644 index 0000000000..8980a57dd6 --- /dev/null +++ b/packages/eve/extension-contracts/reports/dynamicTool/v21.json @@ -0,0 +1,13 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "dynamicTool", + "epoch": 21, + "sha256": "f1a8b3381c4e18f4d34ef1d357849334ded840efc5c6817473c3d0982227ea9c", + "exports": [ + "DynamicToolEntry", + "DynamicToolEvents", + "DynamicToolResult", + "DynamicToolSet", + "defineDynamic" + ] +} diff --git a/packages/eve/extension-contracts/reports/hook/v16.json b/packages/eve/extension-contracts/reports/hook/v16.json new file mode 100644 index 0000000000..68497fe809 --- /dev/null +++ b/packages/eve/extension-contracts/reports/hook/v16.json @@ -0,0 +1,7 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "hook", + "epoch": 16, + "sha256": "8175a76ef37dacf29c7a9c6d50abc146befe42203e97afac55c3853673aa3ac4", + "exports": ["defineHook"] +} diff --git a/packages/eve/extension-contracts/reports/schedule/v5.json b/packages/eve/extension-contracts/reports/schedule/v5.json new file mode 100644 index 0000000000..eb922dd35e --- /dev/null +++ b/packages/eve/extension-contracts/reports/schedule/v5.json @@ -0,0 +1,14 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "schedule", + "epoch": 5, + "sha256": "f630b8248b985772e86a0121dce8bdeabf66f387e2b2c715b56df67ab1c63fad", + "exports": [ + "ScheduleDefinition", + "ScheduleHandlerArgs", + "ScheduleRunHandler", + "ScheduleToFn", + "TypedReceiveTarget", + "defineSchedule" + ] +} diff --git a/packages/eve/extension-contracts/reports/subagent/v4.json b/packages/eve/extension-contracts/reports/subagent/v4.json index 1afb0b48f8..33940b5c8e 100644 --- a/packages/eve/extension-contracts/reports/subagent/v4.json +++ b/packages/eve/extension-contracts/reports/subagent/v4.json @@ -8,6 +8,7 @@ "AgentDefinition", "AgentModelDefinition", "AgentStaticModelDefinition", + "DefinedAgent", "DynamicLocalSubagentDefinition", "DynamicSubagentDefinition", diff --git a/packages/eve/extension-contracts/reports/subagent/v5.json b/packages/eve/extension-contracts/reports/subagent/v5.json index fa955366e0..7e92e58ab9 100644 --- a/packages/eve/extension-contracts/reports/subagent/v5.json +++ b/packages/eve/extension-contracts/reports/subagent/v5.json @@ -8,6 +8,7 @@ "AgentDefinition", "AgentModelDefinition", "AgentStaticModelDefinition", + "DefinedAgent", "DynamicLocalSubagentDefinition", "DynamicSubagentDefinition", diff --git a/packages/eve/extension-contracts/reports/subagent/v6.json b/packages/eve/extension-contracts/reports/subagent/v6.json new file mode 100644 index 0000000000..02a6491a32 --- /dev/null +++ b/packages/eve/extension-contracts/reports/subagent/v6.json @@ -0,0 +1,21 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "subagent", + "epoch": 6, + "sha256": "3169d50307ea907ece054e5c2279ddd1d1ef9a0b69d2a0be37912e659ffed79f", + "exports": [ + "AgentCompactionDefinition", + "AgentDefinition", + "AgentModelDefinition", + "AgentStaticModelDefinition", + "AgentToolOutputDefinition", + "DefinedAgent", + "DynamicLocalSubagentDefinition", + "DynamicSubagentDefinition", + "RemoteAgentDefinition", + "RemoteAgentDefinitionInput", + "defineAgent", + "defineDynamic", + "defineRemoteAgent" + ] +} diff --git a/packages/eve/src/client/index.ts b/packages/eve/src/client/index.ts index 894fc63406..d269903d8f 100644 --- a/packages/eve/src/client/index.ts +++ b/packages/eve/src/client/index.ts @@ -130,6 +130,7 @@ export type { SubagentChildEventStreamEvent, SubagentCompletedStreamEvent, SubagentStartedStreamEvent, + ToolOutputSpilledStreamEvent, TurnCancelledStreamEvent, TurnCompletedStreamEvent, TurnFailedStreamEvent, diff --git a/packages/eve/src/compiler/extension-compatibility.ts b/packages/eve/src/compiler/extension-compatibility.ts index 46d17265ce..f8996ce4b8 100644 --- a/packages/eve/src/compiler/extension-compatibility.ts +++ b/packages/eve/src/compiler/extension-compatibility.ts @@ -27,15 +27,15 @@ const EXTENSION_CAPABILITY_CONTRACTS = { dropped: { 15: "TaskExec replaces stageEffect with send" }, }, dynamicTool: { - current: 20, - supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], + current: 21, + supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21], dropped: {}, }, - channel: { current: 10, supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dropped: {} }, - schedule: { current: 4, supported: [1, 2, 3, 4], dropped: {} }, + channel: { current: 11, supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], dropped: {} }, + schedule: { current: 5, supported: [1, 2, 3, 4, 5], dropped: {} }, subagent: { - current: 5, - supported: [3, 4, 5], + current: 6, + supported: [3, 4, 5, 6], dropped: { 1: "Persistent subagent sessions are now the default and the experimental opt-in was removed", 2: "Persistent subagent sessions are now the default and the experimental opt-in was removed", @@ -47,8 +47,8 @@ const EXTENSION_CAPABILITY_CONTRACTS = { dropped: {}, }, hook: { - current: 15, - supported: [10, 11, 12, 13, 14, 15], + current: 16, + supported: [10, 11, 12, 13, 14, 15, 16], dropped: { 1: "Model identity moved from session.started runtime metadata to step.started call attribution.", 2: "Model identity moved from session.started runtime metadata to step.started call attribution.", @@ -63,14 +63,14 @@ const EXTENSION_CAPABILITY_CONTRACTS = { }, skill: { current: 1, supported: [1], dropped: {} }, dynamicSkill: { - current: 13, - supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], + current: 14, + supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], dropped: {}, }, instructions: { current: 2, supported: [1, 2], dropped: {} }, dynamicInstructions: { - current: 14, - supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], + current: 15, + supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], dropped: {}, }, config: { current: 1, supported: [1], dropped: {} }, diff --git a/packages/eve/src/compiler/manifest.ts b/packages/eve/src/compiler/manifest.ts index f46ca795f4..cb4da2fc38 100644 --- a/packages/eve/src/compiler/manifest.ts +++ b/packages/eve/src/compiler/manifest.ts @@ -54,7 +54,7 @@ export const ROOT_COMPILED_AGENT_NODE_ID = "__root__"; /** * Current compiled manifest schema version. */ -export const COMPILED_AGENT_MANIFEST_VERSION = 44; +export const COMPILED_AGENT_MANIFEST_VERSION = 45; /** * Compiled channel entry preserved in the compiled manifest. @@ -571,6 +571,13 @@ const compiledAgentCompactionDefinitionSchema: z.ZodType = z.union([ @@ -1160,6 +1168,13 @@ function cloneCompiledAgentDefinition(config: CompiledAgentDefinition): Compiled maxOutputTokensPerSession: config.limits.maxOutputTokensPerSession, sessionTimeoutMs: config.limits.sessionTimeoutMs, }, + toolOutput: + config.toolOutput === undefined + ? undefined + : { + maxInlineBytes: config.toolOutput.maxInlineBytes, + overflow: config.toolOutput.overflow, + }, source: { ...config.source }, }; diff --git a/packages/eve/src/compiler/normalize-agent-config.ts b/packages/eve/src/compiler/normalize-agent-config.ts index 213d341fd8..ea2e364217 100644 --- a/packages/eve/src/compiler/normalize-agent-config.ts +++ b/packages/eve/src/compiler/normalize-agent-config.ts @@ -89,6 +89,7 @@ export async function compileAgentConfig( reasoning?: CompiledAgentDefinition["reasoning"]; source: ModuleSourceRef; limits?: CompiledAgentDefinition["limits"]; + toolOutput?: CompiledAgentDefinition["toolOutput"]; } = { compaction, name: manifest.agentId, @@ -140,6 +141,12 @@ export async function compileAgentConfig( }; } + if (definition.toolOutput !== undefined) { + compiledConfig.toolOutput = { + maxInlineBytes: definition.toolOutput.maxInlineBytes, + overflow: definition.toolOutput.overflow, + }; + } if (definition.compaction?.model !== undefined) { compaction.model = await normalizeAuthoredModelReference({ modelCatalog: context.modelCatalog, diff --git a/packages/eve/src/compiler/normalize-manifest.test.ts b/packages/eve/src/compiler/normalize-manifest.test.ts index 02aa06fa76..64e3dc4e4e 100644 --- a/packages/eve/src/compiler/normalize-manifest.test.ts +++ b/packages/eve/src/compiler/normalize-manifest.test.ts @@ -66,6 +66,10 @@ describe("compileAgentManifest source graph", () => { loadNamespace: async () => ({ default: defineAgent({ model: "openai/gpt-5.4", + toolOutput: { + maxInlineBytes: 64 * 1024, + overflow: "sandbox", + }, }), }), }, @@ -87,6 +91,10 @@ describe("compileAgentManifest source graph", () => { const weather = compiled.tools.find((tool) => tool.name === "weather"); expect(compiled.config.source.logicalPath).toBe("agent.ts"); + expect(compiled.config.toolOutput).toEqual({ + maxInlineBytes: 64 * 1024, + overflow: "sandbox", + }); expect(compiled.bindings[compiled.config.source.sourceId]?.owner).toEqual({ kind: "application", }); diff --git a/packages/eve/src/context/keys.ts b/packages/eve/src/context/keys.ts index 332c4bd7a6..c1f54455d7 100644 --- a/packages/eve/src/context/keys.ts +++ b/packages/eve/src/context/keys.ts @@ -21,6 +21,7 @@ import type { import { ContextKey } from "#context/key.js"; import { SESSION_CALLBACK_CONTEXT_KEY_NAME } from "#context/key-names.js"; import type { InstrumentationChannelDeliveryRef } from "#harness/instrumentation/lifecycle.js"; +import type { ToolOutputSpill } from "#harness/tool-output-overflow.js"; import type { HandleEventFn } from "#harness/types.js"; import type { DurableDynamicToolCallbacks } from "#tools/durable-callbacks.js"; import type { DynamicSubagentAgentConfig } from "#runtime/subagents/dynamic-agent-config.js"; @@ -132,6 +133,9 @@ export const SessionCallbackKey = new ContextKey( export const SessionKey = new ContextKey("eve.session"); export const SandboxKey = new ContextKey("eve.sandbox"); +export const PendingToolOutputSpillsKey = new ContextKey( + "eve.pendingToolOutputSpills", +); export const HandleEventKey = new ContextKey("eve.internal.handleEvent"); // --------------------------------------------------------------------------- diff --git a/packages/eve/src/execution/effective-agent-config.test.ts b/packages/eve/src/execution/effective-agent-config.test.ts index 368b4f449e..891e4c8405 100644 --- a/packages/eve/src/execution/effective-agent-config.test.ts +++ b/packages/eve/src/execution/effective-agent-config.test.ts @@ -16,6 +16,7 @@ describe("resolveEffectiveAgentRuntime", () => { limits: { sessionTimeoutMs: 120_000 }, model: { id: "anthropic/claude-opus-4.6" }, reasoning: "high", + toolOutput: { maxInlineBytes: 65_536, overflow: "sandbox" }, }); const tools = [{ name: "search" }]; @@ -25,6 +26,7 @@ describe("resolveEffectiveAgentRuntime", () => { config: { compaction: { thresholdPercent: 0.9 }, limits: { sessionTimeoutMs: 60_000 }, + toolOutput: { maxInlineBytes: 131_072, overflow: "sandbox" }, }, }, turnAgent: { @@ -41,6 +43,7 @@ describe("resolveEffectiveAgentRuntime", () => { expect(effective).toMatchObject({ limits: { sessionTimeoutMs: 120_000 }, thresholdPercent: 0.75, + toolOutput: { maxInlineBytes: 65_536, overflow: "sandbox" }, turnAgent: { compactionModel: { id: "anthropic/claude-sonnet-4.5" }, model: { id: "anthropic/claude-opus-4.6" }, diff --git a/packages/eve/src/execution/effective-agent-config.ts b/packages/eve/src/execution/effective-agent-config.ts index c9a099bdde..9ec234fab2 100644 --- a/packages/eve/src/execution/effective-agent-config.ts +++ b/packages/eve/src/execution/effective-agent-config.ts @@ -3,11 +3,13 @@ import { DynamicSubagentAgentConfigKey } from "#context/keys.js"; import type { RuntimeTurnAgent } from "#runtime/agent/bootstrap.js"; import type { CompiledBundle } from "#runtime/sessions/runtime-context-keys.js"; import type { AgentLimitsDefinition } from "#shared/agent-definition.js"; +import type { AgentToolOutputDefinition } from "#shared/agent-definition.js"; import type { DynamicSubagentAgentConfig } from "#runtime/subagents/dynamic-agent-config.js"; export interface EffectiveAgentRuntime { readonly limits?: AgentLimitsDefinition; readonly thresholdPercent?: number; + readonly toolOutput?: AgentToolOutputDefinition; readonly turnAgent: RuntimeTurnAgent; } @@ -29,6 +31,7 @@ export function resolveEffectiveAgentRuntimeFromConfig( return { limits: bundle.resolvedAgent.config?.limits, thresholdPercent: bundle.resolvedAgent.config?.compaction?.thresholdPercent, + toolOutput: bundle.resolvedAgent.config?.toolOutput, turnAgent: bundle.turnAgent, }; } @@ -43,6 +46,7 @@ export function resolveEffectiveAgentRuntimeFromConfig( return { limits: config.limits, thresholdPercent: config.compaction?.thresholdPercent, + toolOutput: config.toolOutput, turnAgent: { ...turnAgent, compactionModel: config.compaction?.model, diff --git a/packages/eve/src/execution/node-step.ts b/packages/eve/src/execution/node-step.ts index f1b708cb32..e359a75b36 100644 --- a/packages/eve/src/execution/node-step.ts +++ b/packages/eve/src/execution/node-step.ts @@ -77,6 +77,8 @@ export interface CreateExecutionNodeStepInput { readonly mode: RunMode; readonly modelResolutionScope: RuntimeModelResolutionScope; readonly node: ResolvedRuntimeAgentNode; + /** Effective model-facing tool output policy for this agent selection. */ + readonly toolOutput?: NonNullable["toolOutput"]; /** * Effective `maxSubagents` cap configured by the experimental Workflow tool * definition and materialized on the session at creation. @@ -117,6 +119,9 @@ export function createExecutionNodeStep(input: CreateExecutionNodeStepInput): St resolveModel, runtimeIdentity: buildRuntimeIdentity(input.node), tools, + toolOutput: Object.hasOwn(input, "toolOutput") + ? input.toolOutput + : input.node.agent.config?.toolOutput, }); if (instrumentation === undefined) return step; return async (session, stepInput) => { diff --git a/packages/eve/src/execution/workflow-steps.ts b/packages/eve/src/execution/workflow-steps.ts index aa514eb17c..22dca1e1b6 100644 --- a/packages/eve/src/execution/workflow-steps.ts +++ b/packages/eve/src/execution/workflow-steps.ts @@ -86,10 +86,7 @@ import { createDurableSessionState, readDurableSession } from "#execution/durabl import type { TurnStepInput } from "#execution/durable-session-migrations/turn-workflow.js"; import { buildRuntimeIdentity, createExecutionNodeStep } from "#execution/node-step.js"; import { appendTaskAgentAnnouncement } from "#execution/tasks/parent/agent-views.js"; -import { - resolveInitiatingTaskContext, - resolveTaskDeliveryContext, -} from "#tasks/delivery-context.js"; +import { prepareTaskDeliveryModelContext } from "#tasks/delivery-context.js"; import { readRetainedBackgroundToolResult, runBackgroundStep, @@ -254,35 +251,6 @@ export async function turnStep(rawInput: TurnStepInput): Promise { + resolved = await prepareTaskDeliveryModelContext({ + ctx, + policy: effectiveAgent.toolOutput, + resolved, + state: durableSession.state, + taskDeliveryId: + rawInput.input?.kind === "deliver" ? rawInput.input.taskDeliveryId : undefined, + tasksEnabled, + turnId: activeTurnId(initialEmissionState), + }); + ctx.setVirtualContext(HandleEventKey, handleEvent); let schemaSession = resolveEffectiveOutputSchema({ agentOutputSchema: effectiveAgent.turnAgent.outputSchema, @@ -533,6 +512,7 @@ export async function turnStep(rawInput: TurnStepInput): Promise>; } +export interface InstrumentationToolOutputSpilledEvent { + readonly type: "tool.output.spilled"; + readonly bytes: number; + readonly callId: string; + readonly idempotencyKey: string; + readonly maxInlineBytes: number; + readonly path: string; + readonly sequence: number; + readonly sessionId: string; + readonly spillId: string; + readonly stepIndex: number; + readonly toolName: string; + readonly turnId: string; +} + export interface InstrumentationModelCallStartedEvent { readonly type: "model.call.started"; readonly idempotencyKey: string; @@ -562,6 +581,7 @@ export interface InstrumentationProviderDefinition { readonly "tool.call.started"?: InstrumentationEventHandler; readonly "tool.call.completed"?: InstrumentationEventHandler; readonly "tool.call.failed"?: InstrumentationEventHandler; + readonly "tool.output.spilled"?: InstrumentationEventHandler; readonly "turn.cancelled"?: InstrumentationEventHandler; readonly "turn.completed"?: InstrumentationEventHandler; readonly "turn.failed"?: InstrumentationEventHandler; @@ -607,6 +627,7 @@ export type InstrumentationPointEvent = | InstrumentationSessionStartedEvent | InstrumentationSessionTransitionEvent | InstrumentationTurnStartedEvent + | InstrumentationToolOutputSpilledEvent | InstrumentationTurnTerminalEvent; export type InstrumentationEvent = InstrumentationCorrelatedEvent | InstrumentationPointEvent; diff --git a/packages/eve/src/harness/instrumentation/native-events.test.ts b/packages/eve/src/harness/instrumentation/native-events.test.ts index 86dbdff129..674d84db35 100644 --- a/packages/eve/src/harness/instrumentation/native-events.test.ts +++ b/packages/eve/src/harness/instrumentation/native-events.test.ts @@ -15,6 +15,7 @@ import { createTurnCancelledEvent, createTurnFailedEvent, createTurnStartedEvent, + createToolOutputSpilledEvent, } from "#protocol/message.js"; import { createInstrumentationHandleEvent, @@ -29,6 +30,7 @@ import { inputIdempotencyKey, sessionIdempotencyKey, turnIdempotencyKey, + toolOutputSpillIdempotencyKey, } from "#harness/instrumentation/lifecycle.js"; import { RuntimeActionSettlementTimesKey } from "#harness/runtime-action-settlement-state.js"; @@ -174,6 +176,51 @@ describe("createInstrumentationHandleEvent", () => { ]); }); + it("publishes tool-output spills as metadata-only lifecycle events", async () => { + const events: InstrumentationEvent[] = []; + const handleEvent = createInstrumentationHandleEvent({ + handleEvent: async () => {}, + hooks: { + capturesContent: false, + publish: async (event) => { + events.push(event); + }, + }, + sessionId: "session-1", + })!; + + await handleEvent( + createToolOutputSpilledEvent({ + bytes: 128, + callId: "call-1", + maxInlineBytes: 32, + path: "/workspace/.eve/tool-results/abc.json", + sequence: 0, + spillId: "abc", + stepIndex: 1, + toolName: "search", + turnId: "turn-1", + }), + ); + + expect(events).toEqual([ + { + bytes: 128, + callId: "call-1", + idempotencyKey: toolOutputSpillIdempotencyKey("session-1", "abc"), + maxInlineBytes: 32, + path: "/workspace/.eve/tool-results/abc.json", + sequence: 0, + sessionId: "session-1", + spillId: "abc", + stepIndex: 1, + toolName: "search", + turnId: "turn-1", + type: "tool.output.spilled", + }, + ]); + }); + it("carries the dispatch lineage onto every turn a child session starts", async () => { const events: { readonly type: string }[] = []; const parentLineage = { diff --git a/packages/eve/src/harness/instrumentation/native-events.ts b/packages/eve/src/harness/instrumentation/native-events.ts index 3a5df1c9d7..be2cbb48ce 100644 --- a/packages/eve/src/harness/instrumentation/native-events.ts +++ b/packages/eve/src/harness/instrumentation/native-events.ts @@ -18,6 +18,7 @@ import { inputIdempotencyKey, sessionIdempotencyKey, turnIdempotencyKey, + toolOutputSpillIdempotencyKey, } from "#harness/instrumentation/lifecycle.js"; import { rememberInstrumentationActionScope, @@ -318,6 +319,13 @@ function toLifecycleEvent( turnId: activeTurnId, type: "session.failed", }; + case "tool.output.spilled": + return { + ...event.data, + idempotencyKey: toolOutputSpillIdempotencyKey(input.sessionId, event.data.spillId), + sessionId: input.sessionId, + type: "tool.output.spilled", + }; case "turn.started": return { idempotencyKey: turnIdempotencyKey(input.sessionId, event.data.turnId), diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.ts index 87d09779b4..065e00824d 100644 --- a/packages/eve/src/harness/tool-loop.test.ts +++ b/packages/eve/src/harness/tool-loop.test.ts @@ -20,6 +20,7 @@ import { InitiatorAuthKey, LiveStepDynamicModelSelectionKey, ParentSessionKey, + PendingToolOutputSpillsKey, SandboxKey, ScheduleIdKey, SessionTraceSeedKey, @@ -65,6 +66,7 @@ import { AGENT_HANDLES_STATE_KEY } from "#harness/handles/store.js"; import { BackgroundToolExecutorKey } from "#harness/background-tools.js"; import { stashToolInterrupt } from "#harness/tool-interrupts.js"; import { appendMissingToolResultMessages, createToolLoopHarness } from "#harness/tool-loop.js"; +import { TOOL_OUTPUT_FILE_REFERENCE_KIND } from "#harness/tool-output-overflow.js"; import { isSessionLimitDecline, TurnCancelledError } from "#harness/turn-cancellation.js"; import { getSessionTokenLimitViolation, @@ -3288,6 +3290,184 @@ describe("createToolLoopHarness", () => { }); }); + it("spills model history before persisting the continuation and emits the full action.result", async () => { + const fullOutput = "contract metadata ".repeat(16); + setupMockAgent({ + finishReason: "tool-calls", + response: { + messages: [ + { + content: [ + { + input: { address: "0x1" }, + toolCallId: "call-overflow", + toolName: "add", + type: "tool-call", + }, + ], + role: "assistant", + }, + { + content: [ + { + output: fullOutput, + toolCallId: "call-overflow", + toolName: "add", + type: "tool-result", + }, + ], + role: "tool", + }, + ], + }, + text: "", + toolCalls: [ + { + input: { address: "0x1" }, + toolCallId: "call-overflow", + toolName: "add", + type: "tool-call", + }, + ], + toolResults: [ + { + input: { address: "0x1" }, + output: fullOutput, + toolCallId: "call-overflow", + toolName: "add", + type: "tool-result", + }, + ], + }); + + const sandbox = mockSandbox(); + const ctx = new ContextContainer(); + ctx.set(SandboxKey, sandbox.access); + const { emit, events } = createEventCollector(); + const runStep = createToolLoopHarness( + createTestConfig("conversation", emit, { + toolOutput: { maxInlineBytes: 32, overflow: "sandbox" }, + }), + ); + const first = await contextStorage.run(ctx, () => + runStep(createTestSession(), { message: "Load the metadata." }), + ); + + expect(events.find((event) => event.type === "action.result")?.data).toMatchObject({ + result: { callId: "call-overflow", output: fullOutput, toolName: "add" }, + }); + expect(events.find((event) => event.type === "tool.output.spilled")?.data).toMatchObject({ + bytes: Buffer.byteLength(fullOutput, "utf8"), + callId: "call-overflow", + maxInlineBytes: 32, + path: sandbox.writes[0]?.path, + sequence: 0, + spillId: expect.stringMatching(/^[a-f0-9]{64}$/u), + stepIndex: 0, + toolName: "add", + turnId: "turn_0", + }); + const checkpointToolMessage = first.session.history.find((message) => message.role === "tool"); + if (checkpointToolMessage?.role !== "tool") { + throw new Error("Expected the continuation session to retain projected tool history."); + } + const checkpointOverflowPart = checkpointToolMessage.content.find( + (part) => part.type === "tool-result", + ); + + expect(sandbox.writes).toHaveLength(1); + expect(sandbox.writes[0]?.content).toBe(fullOutput); + expect(checkpointOverflowPart).toMatchObject({ + output: { + type: "json", + value: { + bytes: Buffer.byteLength(fullOutput, "utf8"), + kind: TOOL_OUTPUT_FILE_REFERENCE_KIND, + path: sandbox.writes[0]?.path, + toolName: "add", + }, + }, + toolCallId: "call-overflow", + toolName: "add", + }); + const next = first.next; + expect(typeof next).toBe("function"); + if (typeof next !== "function") { + throw new Error("Expected the oversized tool result to continue the model loop."); + } + + setupMockAgentError(new Error("Model blew up")); + const failed = await contextStorage.run(ctx, () => next(first.session)); + + const agent = vi.mocked(ToolLoopAgent).mock.results.at(-1)?.value; + if (agent === undefined) { + throw new Error("Expected a second ToolLoopAgent instance."); + } + const modelMessages = vi.mocked(agent.stream).mock.calls[0]?.[0].messages as + | ModelMessage[] + | undefined; + if (modelMessages === undefined) { + throw new Error("Expected the second ToolLoopAgent instance to stream."); + } + const toolMessage = modelMessages.find((message) => message.role === "tool"); + if (toolMessage?.role !== "tool") { + throw new Error("Expected the second model call to receive tool history."); + } + const overflowPart = toolMessage.content.find((part) => part.type === "tool-result"); + const persistedToolMessage = failed.session.history.find((message) => message.role === "tool"); + if (persistedToolMessage?.role !== "tool") { + throw new Error("Expected failed model call history to retain the tool result reference."); + } + const persistedOverflowPart = persistedToolMessage.content.find( + (part) => part.type === "tool-result", + ); + + expect(sandbox.writes).toHaveLength(1); + expect(overflowPart).toEqual(checkpointOverflowPart); + expect(persistedOverflowPart).toEqual(checkpointOverflowPart); + expect(events.filter((event) => event.type === "tool.output.spilled")).toHaveLength(1); + }); + + it("emits task output spills staged before the model step", async () => { + setupMockAgent({ + finishReason: "stop", + response: { messages: [{ content: "done", role: "assistant" }] }, + text: "done", + toolCalls: [], + toolResults: [], + }); + const ctx = new ContextContainer(); + ctx.set(PendingToolOutputSpillsKey, [ + { + bytes: 128, + callId: "task:task_1", + maxInlineBytes: 32, + path: "/workspace/.eve/tool-results/abc.json", + spillId: "abc", + toolName: "task", + }, + ]); + const { emit, events } = createEventCollector(); + const runStep = createToolLoopHarness(createTestConfig("conversation", emit)); + + await contextStorage.run(ctx, () => + runStep(createTestSession(), { message: "Report the task result." }), + ); + + expect(events.find((event) => event.type === "tool.output.spilled")?.data).toEqual({ + bytes: 128, + callId: "task:task_1", + maxInlineBytes: 32, + path: "/workspace/.eve/tool-results/abc.json", + sequence: 0, + spillId: "abc", + stepIndex: 0, + toolName: "task", + turnId: "turn_0", + }); + expect(ctx.get(PendingToolOutputSpillsKey)).toEqual([]); + }); + it("skips AI-SDK-marked invalid tool calls so a malformed JSON payload does not crash the harness", async () => { // Simulates the AI SDK fallback path: when the model emits unparsable // JSON for a tool call, `parseToolCall` returns a DynamicToolCall with diff --git a/packages/eve/src/harness/tool-loop.ts b/packages/eve/src/harness/tool-loop.ts index e944cef856..434dcffde1 100644 --- a/packages/eve/src/harness/tool-loop.ts +++ b/packages/eve/src/harness/tool-loop.ts @@ -35,9 +35,11 @@ import { AuthKey, ChannelInstrumentationKey, ParentSessionKey, + PendingToolOutputSpillsKey, ParentTraceContextKey, ScheduleIdKey, SessionCallbackKey, + SandboxKey, SessionTraceSeedKey, TurnTaskDeliveryKey, TurnTaskStateKey, @@ -75,6 +77,7 @@ import { createInputRequestedEvent, createResultCompletedEvent, createSessionWaitingEvent, + createToolOutputSpilledEvent, type UnstampedMessageStreamEvent, } from "#protocol/message.js"; import type { RuntimeTraceContext } from "#protocol/message.js"; @@ -90,6 +93,10 @@ import { hydrateSandboxAttachments, stageAttachmentsToSandbox, } from "#harness/attachment-staging.js"; +import { + projectOversizedToolResults, + type ToolOutputSpill, +} from "#harness/tool-output-overflow.js"; import { buildWorkflowHostTools, resolveWorkflowSandboxBridgeRequestLimit, @@ -722,6 +729,25 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { sessionId: session.sessionId, turnId: activeTurnId(emissionState), }); + const onToolOutputSpill = + emit === undefined + ? undefined + : async (spill: ToolOutputSpill): Promise => { + await emit( + createToolOutputSpilledEvent({ + ...spill, + sequence: emissionState.sequence, + stepIndex: emissionState.stepIndex, + turnId: activeTurnId(emissionState), + }), + ); + }; + if (onToolOutputSpill !== undefined && store !== undefined) { + for (const spill of store.get(PendingToolOutputSpillsKey) ?? []) { + await onToolOutputSpill(spill); + } + store.set(PendingToolOutputSpillsKey, []); + } const failModelSelection = async ( error: unknown, failureState: ReturnType, @@ -812,6 +838,15 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { const ctx = contextStorage.getStore(); const resolvedModel = await resolveActiveRuntimeModel({ config, ctx, session }); session = resolvedModel.session; + const overflowMessages = await projectOversizedToolResults({ + messages: session.history, + policy: config.toolOutput, + sandboxAccess: ctx?.get(SandboxKey), + onSpill: onToolOutputSpill, + }); + if (overflowMessages !== session.history) { + session = { ...session, history: [...overflowMessages] }; + } const compacted = await maybeCompact({ abortSignal: config.abortSignal, @@ -1318,6 +1353,24 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { } session = resolvedModel.session; const model = resolvedModel.model; + const durableHistoryLength = + resolvedRuntimeActions.outcome === "resolved" + ? resolvedRuntimeActions.messages.length + instructionMessages.length + : session.history.length; + const overflowMessages = await projectOversizedToolResults({ + messages, + policy: config.toolOutput, + sandboxAccess: ctx?.get(SandboxKey), + onSpill: onToolOutputSpill, + }); + if (overflowMessages !== messages) { + messages = [...overflowMessages]; + session = { + ...session, + history: messages.slice(0, durableHistoryLength), + }; + projectedMessages = projectHistory(messages, session.state); + } const cachePath = detectPromptCachePath(model); const marker = cachePath.kind === "anthropic-direct" ? getAnthropicCacheMarker() : undefined; @@ -1349,7 +1402,6 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { messages = compaction.messages; } projectedMessages = normalizeModelMessages(projectHistory(messages, session.state)); - if (emit) { await emitStepStarted( emit, @@ -1358,6 +1410,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { projectedMessages, ); } + const approvedTools = getApprovedTools(session); const isFirstTurn = emissionState.sequence === 0; @@ -2045,6 +2098,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { config, emit, emissionState, + onToolOutputSpill, delegatedCaller: taskUpdatesEnabled, modelPromptMessageCount: projectedMessages.length, promptMessages: messages, @@ -2607,6 +2661,7 @@ async function handleStepResult(input: { readonly config: ToolLoopHarnessConfig; readonly emit?: ToolLoopHarnessConfig["handleEvent"]; readonly emissionState: ReturnType; + readonly onToolOutputSpill?: (spill: ToolOutputSpill) => void | PromiseLike; readonly delegatedCaller: boolean; readonly modelPromptMessageCount: number; readonly promptMessages: readonly ModelMessage[]; @@ -2657,7 +2712,12 @@ async function handleStepResult(input: { messages: rawResponseMessages, providerExecutedOutcomeIds, }); - const responseMessages = normalizedProviderHistory.messages; + const responseMessages = await projectOversizedToolResults({ + messages: normalizedProviderHistory.messages, + policy: config.toolOutput, + sandboxAccess: contextStorage.getStore()?.get(SandboxKey), + onSpill: input.onToolOutputSpill, + }); const baseSession: HarnessSession = { ...session, diff --git a/packages/eve/src/harness/tool-output-overflow.test.ts b/packages/eve/src/harness/tool-output-overflow.test.ts new file mode 100644 index 0000000000..99585e4abb --- /dev/null +++ b/packages/eve/src/harness/tool-output-overflow.test.ts @@ -0,0 +1,359 @@ +import { Buffer } from "node:buffer"; + +import type { ModelMessage } from "ai"; +import { describe, expect, it } from "vitest"; + +import { + projectOversizedToolResults, + type ToolOutputSpill, + TOOL_OUTPUT_FILE_REFERENCE_KIND, + TOOL_OUTPUT_FILES_ROOT, +} from "#harness/tool-output-overflow.js"; +import { mockSandbox } from "#internal/testing/mocks/mock-sandbox.js"; +import type { AgentToolOutputDefinition } from "#shared/agent-definition.js"; + +const POLICY: AgentToolOutputDefinition = { + maxInlineBytes: 32, + overflow: "sandbox", +}; + +describe("projectOversizedToolResults", () => { + it("preserves current behavior when the policy is absent", async () => { + const sandbox = mockSandbox(); + const messages: ModelMessage[] = [ + { + content: [ + { + output: { type: "text", value: "x".repeat(100) }, + toolCallId: "call-large", + toolName: "fetch", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + + const projected = await projectOversizedToolResults({ + messages, + policy: undefined, + sandboxAccess: sandbox.access, + }); + + expect(projected).toBe(messages); + expect(sandbox.writes).toEqual([]); + }); + + it("keeps small outputs inline without opening the sandbox", async () => { + const sandbox = mockSandbox(); + const messages: ModelMessage[] = [ + { + content: [ + { + output: { type: "text", value: "small" }, + toolCallId: "call-small", + toolName: "fetch", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + + const projected = await projectOversizedToolResults({ + messages, + policy: POLICY, + sandboxAccess: sandbox.access, + }); + + expect(projected).toBe(messages); + expect(sandbox.writes).toEqual([]); + }); + + it("keeps execution denials inline regardless of the byte threshold", async () => { + const sandbox = mockSandbox(); + const messages: ModelMessage[] = [ + { + content: [ + { + output: { + type: "execution-denied", + reason: "Tool execution was denied.", + }, + toolCallId: "call-denied", + toolName: "bash", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + + const projected = await projectOversizedToolResults({ + messages, + policy: { maxInlineBytes: 1, overflow: "sandbox" }, + sandboxAccess: sandbox.access, + }); + + expect(projected).toBe(messages); + expect(sandbox.writes).toEqual([]); + }); + + it("writes oversized discovered MCP JSON and replaces only the model-facing output", async () => { + const sandbox = mockSandbox(); + const output = { + content: [{ text: "x".repeat(80), type: "text" }], + structuredContent: { result: "x".repeat(80) }, + }; + const messages: ModelMessage[] = [ + { + content: [ + { + output: { type: "json", value: output }, + toolCallId: "call-mcp", + toolName: "herd__contractMetadataTool", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + + const spills: ToolOutputSpill[] = []; + const projected = await projectOversizedToolResults({ + messages, + policy: POLICY, + sandboxAccess: sandbox.access, + onSpill: (spill) => { + spills.push(spill); + }, + }); + + expect(sandbox.writes).toHaveLength(1); + expect(sandbox.writes[0]?.content).toBe(JSON.stringify(output, null, 2)); + expect(sandbox.writes[0]?.path).toMatch( + new RegExp(`^${TOOL_OUTPUT_FILES_ROOT}/[a-f0-9]{64}\\.json$`), + ); + expect(projected).toEqual([ + { + content: [ + { + output: { + type: "json", + value: { + bytes: Buffer.byteLength(JSON.stringify(output), "utf8"), + kind: TOOL_OUTPUT_FILE_REFERENCE_KIND, + path: sandbox.writes[0]?.path, + toolName: "herd__contractMetadataTool", + }, + }, + toolCallId: "call-mcp", + toolName: "herd__contractMetadataTool", + type: "tool-result", + }, + ], + role: "tool", + }, + ]); + expect(spills).toEqual([ + { + bytes: Buffer.byteLength(JSON.stringify(output), "utf8"), + callId: "call-mcp", + maxInlineBytes: POLICY.maxInlineBytes, + path: sandbox.writes[0]?.path, + spillId: sandbox.writes[0]?.path.match(/([a-f0-9]{64})\.json$/u)?.[1], + toolName: "herd__contractMetadataTool", + }, + ]); + }); + + it("uses one deterministic text path for replayed call ids", async () => { + const sandbox = mockSandbox(); + const messages: ModelMessage[] = [ + { + content: [ + { + output: { type: "text", value: "full text ".repeat(20) }, + toolCallId: "call-replayed", + toolName: "report", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + + const first = await projectOversizedToolResults({ + messages, + policy: POLICY, + sandboxAccess: sandbox.access, + }); + await projectOversizedToolResults({ + messages, + policy: POLICY, + sandboxAccess: sandbox.access, + }); + const replayedReference = first[0]?.role === "tool" ? first[0].content[0] : undefined; + + expect(sandbox.writes).toHaveLength(2); + expect(sandbox.writes[0]?.path).toBe(sandbox.writes[1]?.path); + expect(sandbox.writes[0]?.path).toMatch(/\.txt$/); + expect(sandbox.writes[0]?.content).toBe("full text ".repeat(20)); + expect(replayedReference?.type).toBe("tool-result"); + }); + + it("does not overwrite a prior result when a call id is reused", async () => { + const sandbox = mockSandbox(); + const message = (value: string): ModelMessage => ({ + content: [ + { + output: { type: "text", value }, + toolCallId: "call-reused", + toolName: "report", + type: "tool-result", + }, + ], + role: "tool", + }); + + await projectOversizedToolResults({ + messages: [message("first ".repeat(20))], + policy: POLICY, + sandboxAccess: sandbox.access, + }); + await projectOversizedToolResults({ + messages: [message("second ".repeat(20))], + policy: POLICY, + sandboxAccess: sandbox.access, + }); + + expect(sandbox.writes).toHaveLength(2); + expect(sandbox.writes[0]?.path).not.toBe(sandbox.writes[1]?.path); + }); + + it("spills lookalike references with additional payload fields", async () => { + const sandbox = mockSandbox(); + const messages: ModelMessage[] = [ + { + content: [ + { + output: { + type: "json", + value: { + bytes: 1000, + extra: "x".repeat(100), + kind: TOOL_OUTPUT_FILE_REFERENCE_KIND, + path: "/workspace/.eve/tool-results/existing.json", + toolName: "fetch", + }, + }, + toolCallId: "call-lookalike", + toolName: "fetch", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + + const projected = await projectOversizedToolResults({ + messages, + policy: POLICY, + sandboxAccess: sandbox.access, + }); + + expect(sandbox.writes).toHaveLength(1); + expect(projected).not.toBe(messages); + }); + + it("keeps connection_search results inline for later dynamic-tool reconstruction", async () => { + const sandbox = mockSandbox(); + const messages: ModelMessage[] = [ + { + content: [ + { + output: { type: "json", value: [{ description: "x".repeat(100) }] }, + toolCallId: "call-search", + toolName: "connection_search", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + + const projected = await projectOversizedToolResults({ + messages, + policy: POLICY, + sandboxAccess: sandbox.access, + }); + + expect(projected).toBe(messages); + expect(sandbox.writes).toEqual([]); + }); + + it("preserves tool error semantics on an oversized error payload", async () => { + const sandbox = mockSandbox(); + const messages: ModelMessage[] = [ + { + content: [ + { + output: { type: "error-text", value: "upstream failure ".repeat(20) }, + toolCallId: "call-error", + toolName: "fetch", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + + const projected = await projectOversizedToolResults({ + messages, + policy: POLICY, + sandboxAccess: sandbox.access, + }); + const part = projected[0]?.role === "tool" ? projected[0].content[0] : undefined; + + expect(part).toMatchObject({ + output: { + type: "error-json", + value: { kind: TOOL_OUTPUT_FILE_REFERENCE_KIND }, + }, + }); + }); + + it("does not spill an existing eve file reference again", async () => { + const sandbox = mockSandbox(); + const messages: ModelMessage[] = [ + { + content: [ + { + output: { + type: "json", + value: { + bytes: 1000, + kind: TOOL_OUTPUT_FILE_REFERENCE_KIND, + path: `/workspace/.eve/tool-results/${"a".repeat(64)}.json`, + toolName: "fetch", + }, + }, + toolCallId: "call-existing", + toolName: "fetch", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + + const projected = await projectOversizedToolResults({ + messages, + policy: { maxInlineBytes: 1, overflow: "sandbox" }, + sandboxAccess: sandbox.access, + }); + + expect(projected).toBe(messages); + expect(sandbox.writes).toEqual([]); + }); +}); diff --git a/packages/eve/src/harness/tool-output-overflow.ts b/packages/eve/src/harness/tool-output-overflow.ts new file mode 100644 index 0000000000..6857eeaf98 --- /dev/null +++ b/packages/eve/src/harness/tool-output-overflow.ts @@ -0,0 +1,260 @@ +import { Buffer } from "node:buffer"; +import { createHash } from "node:crypto"; + +import type { JSONValue, ModelMessage } from "ai"; +import type { SandboxSession } from "#public/definitions/sandbox.js"; + +import type { SandboxAccess } from "#sandbox/state.js"; +import type { JsonObject, JsonValue } from "#shared/json.js"; +import type { AgentToolOutputDefinition } from "#shared/agent-definition.js"; + +export const TOOL_OUTPUT_FILES_ROOT = "/workspace/.eve/tool-results"; +export const TOOL_OUTPUT_FILE_REFERENCE_KIND = "eve-tool-output-file"; +const TOOL_OUTPUT_FILE_REFERENCE_PATH = + /^\/workspace\/\.eve\/tool-results\/[a-f0-9]{64}\.(?:json|txt)$/; + +type ToolResponsePart = Extract["content"][number]; +type ToolResultPart = Extract; + +export interface ToolOutputSpill { + readonly bytes: number; + readonly callId: string; + readonly maxInlineBytes: number; + readonly path: string; + readonly spillId: string; + readonly toolName: string; +} + +type SerializedToolOutput = + | { + readonly bytes: number; + readonly content: string; + readonly extension: "txt"; + } + | { + readonly bytes: number; + readonly compact: string; + readonly extension: "json"; + readonly value: unknown; + }; + +/** + * Replaces oversized model-facing tool results with durable sandbox file references. + * Framework control results remain inline because later steps reconstruct state from them. + */ +export async function projectOversizedToolResults(input: { + readonly messages: readonly ModelMessage[]; + readonly policy: AgentToolOutputDefinition | undefined; + readonly sandboxAccess: SandboxAccess | undefined; + readonly onSpill?: (spill: ToolOutputSpill) => void | PromiseLike; +}): Promise { + if (input.policy === undefined) { + return input.messages; + } + + let sandbox: SandboxSession | undefined; + let changed = false; + const projected: ModelMessage[] = []; + + for (const message of input.messages) { + if (message.role !== "tool" || !Array.isArray(message.content)) { + projected.push(message); + continue; + } + + let messageChanged = false; + const content: ToolResponsePart[] = []; + for (const part of message.content) { + if ( + part.type !== "tool-result" || + part.toolName === "connection_search" || + (typeof part.output === "object" && + part.output !== null && + "type" in part.output && + part.output.type === "execution-denied") || + isToolOutputFileReference(part.output) + ) { + content.push(part); + continue; + } + + const serialized = serializeToolOutput(part.output); + if (serialized === null || serialized.bytes <= input.policy.maxInlineBytes) { + content.push(part); + continue; + } + + if (sandbox === undefined) { + if (input.sandboxAccess === undefined) { + throw new Error( + "Agent tool-output overflow is configured, but sandbox access is unavailable for this step.", + ); + } + const activeSandbox = await input.sandboxAccess.get(); + if (activeSandbox === null) { + throw new Error( + "Agent tool-output overflow is configured, but this session has no active sandbox.", + ); + } + sandbox = activeSandbox; + } + + const digest = createHash("sha256") + .update(part.toolCallId) + .update("\0") + .update(serialized.extension === "txt" ? serialized.content : serialized.compact) + .digest("hex"); + const authoredPath = `${TOOL_OUTPUT_FILES_ROOT}/${digest}.${serialized.extension}`; + const fileContent = + serialized.extension === "txt" + ? serialized.content + : (JSON.stringify(serialized.value, null, 2) ?? serialized.compact); + await sandbox.writeTextFile({ content: fileContent, path: authoredPath }); + const reference = { + bytes: serialized.bytes, + kind: TOOL_OUTPUT_FILE_REFERENCE_KIND, + path: sandbox.resolvePath(authoredPath), + toolName: part.toolName, + } satisfies JsonObject; + await input.onSpill?.({ + bytes: serialized.bytes, + callId: part.toolCallId, + maxInlineBytes: input.policy.maxInlineBytes, + path: reference.path, + spillId: digest, + toolName: part.toolName, + }); + content.push({ + ...part, + output: { + type: + typeof part.output === "object" && + part.output !== null && + "type" in part.output && + (part.output.type === "error-json" || part.output.type === "error-text") + ? "error-json" + : "json", + value: reference, + }, + }); + changed = true; + messageChanged = true; + } + + projected.push(messageChanged ? { ...message, content } : message); + } + + return changed ? projected : input.messages; +} + +/** Projects one task-delivered value through the agent's tool-output policy. */ +export async function projectOversizedToolOutputValue(input: { + readonly output: JsonValue; + readonly policy: AgentToolOutputDefinition | undefined; + readonly sandboxAccess: SandboxAccess | undefined; + readonly taskId: string; + readonly onSpill?: (spill: ToolOutputSpill) => void | PromiseLike; +}): Promise { + const messages: ModelMessage[] = [ + { + content: [ + { + output: { type: "json", value: input.output as JSONValue }, + toolCallId: `task:${input.taskId}`, + toolName: "task", + type: "tool-result", + }, + ], + role: "tool", + }, + ]; + const projected = await projectOversizedToolResults({ + messages, + policy: input.policy, + sandboxAccess: input.sandboxAccess, + onSpill: input.onSpill, + }); + const message = projected[0]; + const part = message?.role === "tool" ? message.content[0] : undefined; + return part?.type === "tool-result" && + typeof part.output === "object" && + part.output !== null && + "value" in part.output + ? (part.output.value as JsonValue) + : input.output; +} + +function serializeToolOutput(output: ToolResultPart["output"]): SerializedToolOutput | null { + if (typeof output === "string") { + return { + bytes: Buffer.byteLength(output, "utf8"), + content: output, + extension: "txt", + }; + } + + if (typeof output === "object" && output !== null && "type" in output && "value" in output) { + const type = output.type; + const value = output.value; + if ((type === "text" || type === "error-text") && typeof value === "string") { + return { + bytes: Buffer.byteLength(value, "utf8"), + content: value, + extension: "txt", + }; + } + if (type === "json" || type === "error-json") { + return jsonOutput(value); + } + } + + return jsonOutput(output); +} + +function jsonOutput(value: unknown): SerializedToolOutput | null { + const compact = JSON.stringify(value); + if (compact === undefined) { + return null; + } + return { + bytes: Buffer.byteLength(compact, "utf8"), + compact, + extension: "json", + value, + }; +} + +function isToolOutputFileReference(output: ToolResultPart["output"]): boolean { + if ( + typeof output !== "object" || + output === null || + !("type" in output) || + !("value" in output) + ) { + return false; + } + if ( + (output.type !== "json" && output.type !== "error-json") || + typeof output.value !== "object" || + output.value === null + ) { + return false; + } + const value = output.value as Record; + const keys = Object.keys(value).sort(); + return ( + keys.length === 4 && + keys[0] === "bytes" && + keys[1] === "kind" && + keys[2] === "path" && + keys[3] === "toolName" && + typeof value.bytes === "number" && + Number.isSafeInteger(value.bytes) && + value.bytes >= 0 && + value.kind === TOOL_OUTPUT_FILE_REFERENCE_KIND && + typeof value.path === "string" && + TOOL_OUTPUT_FILE_REFERENCE_PATH.test(value.path) && + typeof value.toolName === "string" && + value.toolName.length > 0 + ); +} diff --git a/packages/eve/src/harness/types.ts b/packages/eve/src/harness/types.ts index 54e0c77ed0..ad1376f06b 100644 --- a/packages/eve/src/harness/types.ts +++ b/packages/eve/src/harness/types.ts @@ -12,7 +12,10 @@ import type { JsonObject } from "#shared/json.js"; import type { TokenUsage } from "#shared/token-usage.js"; import type { InternalToolDefinition } from "#tools/definition.js"; import type { WebSearchProvider } from "#shared/web-search.js"; -import type { AgentReasoningDefinition } from "#shared/agent-definition.js"; +import type { + AgentReasoningDefinition, + AgentToolOutputDefinition, +} from "#shared/agent-definition.js"; import type { HarnessToolDefinition } from "#harness/execute-tool.js"; import type { HarnessInstrumentation } from "#harness/instrumentation/runtime.js"; import type { HistoryViewProjector, PreparedHistoryView } from "#shared/history-view.js"; @@ -346,6 +349,8 @@ export interface ToolLoopHarnessConfig { * authoritative server-side metadata. */ readonly runtimeIdentity?: RuntimeIdentity; + /** Optional agent-wide policy for spilling oversized model-facing tool results. */ + readonly toolOutput?: AgentToolOutputDefinition; /** * Unified tool definitions for this harness step. * diff --git a/packages/eve/src/instrumentation/provider.ts b/packages/eve/src/instrumentation/provider.ts index 7c9bbd0d00..096b06c1d8 100644 --- a/packages/eve/src/instrumentation/provider.ts +++ b/packages/eve/src/instrumentation/provider.ts @@ -59,6 +59,7 @@ export type { InstrumentationToolCallFailedEvent, InstrumentationToolCallStartedEvent, InstrumentationToolOutput, + InstrumentationToolOutputSpilledEvent, InstrumentationTraceContext, InstrumentationTurnFailedEvent, InstrumentationTurnSettledEvent, diff --git a/packages/eve/src/internal/authored-definition/core.test.ts b/packages/eve/src/internal/authored-definition/core.test.ts index f37904312f..91ec0b2f08 100644 --- a/packages/eve/src/internal/authored-definition/core.test.ts +++ b/packages/eve/src/internal/authored-definition/core.test.ts @@ -22,6 +22,46 @@ describe("normalizeAgentDefinition", () => { expect(definition.reasoning).toBe("high"); }); + it("accepts a sandbox tool-output overflow policy", () => { + const definition = normalizeAgentDefinition( + { + model: "openai/gpt-5.5", + toolOutput: { + maxInlineBytes: 64 * 1024, + overflow: "sandbox", + }, + }, + FAILURE_MESSAGE, + ); + + expect(definition.toolOutput).toEqual({ + maxInlineBytes: 64 * 1024, + overflow: "sandbox", + }); + }); + + it("rejects incomplete or unsupported tool-output policies", () => { + expect(() => + normalizeAgentDefinition( + { + model: "openai/gpt-5.5", + toolOutput: { overflow: "sandbox" }, + }, + FAILURE_MESSAGE, + ), + ).toThrow('"toolOutput" requires a positive integer "maxInlineBytes"'); + + expect(() => + normalizeAgentDefinition( + { + model: "openai/gpt-5.5", + toolOutput: { maxInlineBytes: 1024, overflow: "truncate" }, + }, + FAILURE_MESSAGE, + ), + ).toThrow('overflow: "sandbox"'); + }); + it("accepts dynamic model definitions", () => { const model = defineDynamic({ events: { diff --git a/packages/eve/src/internal/authored-definition/core.ts b/packages/eve/src/internal/authored-definition/core.ts index 281a0fd92e..c96dd89bb8 100644 --- a/packages/eve/src/internal/authored-definition/core.ts +++ b/packages/eve/src/internal/authored-definition/core.ts @@ -56,6 +56,7 @@ export function normalizeAgentDefinition( "modelOptions", "outputSchema", "reasoning", + "toolOutput", ], message, ); @@ -115,6 +116,20 @@ export function normalizeAgentDefinition( definition.limits = normalizeAgentLimitsDefinition(record.limits, message); } + if (record.toolOutput !== undefined) { + const toolOutput = expectObjectRecord(record.toolOutput, message); + expectOnlyKnownKeys(toolOutput, ["maxInlineBytes", "overflow"], message); + if (toolOutput.maxInlineBytes === undefined || toolOutput.overflow !== "sandbox") { + throw new Error( + `${message} "toolOutput" requires a positive integer "maxInlineBytes" and overflow: "sandbox".`, + ); + } + definition.toolOutput = { + maxInlineBytes: expectPositiveInteger(toolOutput.maxInlineBytes, message), + overflow: "sandbox", + }; + } + return definition as Readonly; } diff --git a/packages/eve/src/protocol/message.test.ts b/packages/eve/src/protocol/message.test.ts index 4a3698579b..bf313ffcb4 100644 --- a/packages/eve/src/protocol/message.test.ts +++ b/packages/eve/src/protocol/message.test.ts @@ -17,6 +17,7 @@ import { createStepStartedEvent, createSubagentCalledEvent, createTurnCancelledEvent, + createToolOutputSpilledEvent, encodeMessageStreamEvent, stampMessageStreamEvent, } from "#protocol/message.js"; @@ -25,7 +26,36 @@ import { createEveConnectionCallbackRoutePath } from "#protocol/routes.js"; describe("message stream protocol", () => { it("pins the stream version for timed session events", () => { - expect(EVE_MESSAGE_STREAM_VERSION).toBe("24"); + expect(EVE_MESSAGE_STREAM_VERSION).toBe("25"); + }); + + it("creates model-facing tool-output spill notifications", () => { + expect( + createToolOutputSpilledEvent({ + bytes: 131_072, + callId: "call-1", + maxInlineBytes: 65_536, + path: "/workspace/.eve/tool-results/abc.json", + sequence: 2, + spillId: "abc", + stepIndex: 1, + toolName: "search", + turnId: "turn_2", + }), + ).toEqual({ + data: { + bytes: 131_072, + callId: "call-1", + maxInlineBytes: 65_536, + path: "/workspace/.eve/tool-results/abc.json", + sequence: 2, + spillId: "abc", + stepIndex: 1, + toolName: "search", + turnId: "turn_2", + }, + type: "tool.output.spilled", + }); }); it("creates authoritative input resolution batches", () => { diff --git a/packages/eve/src/protocol/message.ts b/packages/eve/src/protocol/message.ts index 4a9ab24eda..d0edb82d56 100644 --- a/packages/eve/src/protocol/message.ts +++ b/packages/eve/src/protocol/message.ts @@ -27,7 +27,7 @@ export const EVE_STREAM_TAIL_INDEX_HEADER = "x-eve-stream-tail-index"; export const EVE_STREAM_VERSION_HEADER = "x-eve-stream-version"; export const EVE_MESSAGE_STREAM_CONTENT_TYPE = "application/x-ndjson; charset=utf-8"; export const EVE_MESSAGE_STREAM_FORMAT = "ndjson"; -export const EVE_MESSAGE_STREAM_VERSION = "24"; +export const EVE_MESSAGE_STREAM_VERSION = "25"; /** * eve-owned finish reason for one completed assistant step. @@ -316,6 +316,22 @@ export interface ActionResultStreamEvent { type: "action.result"; } +/** Stream event emitted after eve durably spills model-facing tool output. */ +export interface ToolOutputSpilledStreamEvent { + data: { + bytes: number; + callId: string; + maxInlineBytes: number; + path: string; + sequence: number; + spillId: string; + stepIndex: number; + toolName: string; + turnId: string; + }; + type: "tool.output.spilled"; +} + /** * Stream event emitted for a preliminary snapshot from a locally executed * tool generator. The final snapshot is emitted as `action.result`. @@ -759,6 +775,7 @@ export type UnstampedMessageStreamEvent = | InputResolvedStreamEvent | ActionPartialStreamEvent | ActionResultStreamEvent + | ToolOutputSpilledStreamEvent | ReasoningCompletedStreamEvent | StepCompletedStreamEvent | StepFailedStreamEvent @@ -1293,6 +1310,24 @@ export function createActionResultEvent(input: { }; } +/** Creates a durable notification for one model-facing tool-output spill. */ +export function createToolOutputSpilledEvent(input: { + readonly bytes: number; + readonly callId: string; + readonly maxInlineBytes: number; + readonly path: string; + readonly sequence: number; + readonly spillId: string; + readonly stepIndex: number; + readonly toolName: string; + readonly turnId: string; +}): ToolOutputSpilledStreamEvent { + return { + data: { ...input }, + type: "tool.output.spilled", + }; +} + /** Creates an `action.partial` event for one preliminary tool-result snapshot. */ export function createActionPartialEvent(input: { readonly result: RuntimeToolResultActionResult; diff --git a/packages/eve/src/public/definitions/agent.ts b/packages/eve/src/public/definitions/agent.ts index 04fe990662..fee94900c1 100644 --- a/packages/eve/src/public/definitions/agent.ts +++ b/packages/eve/src/public/definitions/agent.ts @@ -26,6 +26,7 @@ export type { PublicAgentModelDefinition as AgentModelDefinition, PublicAgentStaticModelDefinition as AgentStaticModelDefinition, PublicAgentCompactionDefinition as AgentCompactionDefinition, + AgentToolOutputDefinition, } from "#shared/agent-definition.js"; /** diff --git a/packages/eve/src/public/definitions/hook.ts b/packages/eve/src/public/definitions/hook.ts index 48e2bc3b5a..6960fa4a5a 100644 --- a/packages/eve/src/public/definitions/hook.ts +++ b/packages/eve/src/public/definitions/hook.ts @@ -45,6 +45,7 @@ export interface HookEventMap { readonly "subagent.completed": ProtocolEvent<"subagent.completed">; readonly "subagent.event": ProtocolEvent<"subagent.event">; readonly "subagent.started": ProtocolEvent<"subagent.started">; + readonly "tool.output.spilled": ProtocolEvent<"tool.output.spilled">; readonly "turn.cancelled": ProtocolEvent<"turn.cancelled">; readonly "turn.completed": ProtocolEvent<"turn.completed">; readonly "turn.failed": ProtocolEvent<"turn.failed">; diff --git a/packages/eve/src/public/index.ts b/packages/eve/src/public/index.ts index 3e0191e245..dccc6638cf 100644 --- a/packages/eve/src/public/index.ts +++ b/packages/eve/src/public/index.ts @@ -11,6 +11,7 @@ export { type AgentModelOptionsDefinition, type AgentReasoningDefinition, type AgentStaticModelDefinition, + type AgentToolOutputDefinition, type AgentWorkflowDefinition, type AgentWorkflowWorldDefinition, type DefinedAgent, diff --git a/packages/eve/src/runtime/resolve-agent.ts b/packages/eve/src/runtime/resolve-agent.ts index 33fa7e4187..bf53261138 100644 --- a/packages/eve/src/runtime/resolve-agent.ts +++ b/packages/eve/src/runtime/resolve-agent.ts @@ -188,6 +188,7 @@ function createResolvedAgentConfig( reasoning?: NonNullable["reasoning"]; source?: NonNullable["source"]; limits?: NonNullable["limits"]; + toolOutput?: NonNullable["toolOutput"]; } = { name: manifest.config.name, }; @@ -259,6 +260,13 @@ function createResolvedAgentConfig( }; } + if (manifest.config.toolOutput !== undefined) { + config.toolOutput = { + maxInlineBytes: manifest.config.toolOutput.maxInlineBytes, + overflow: manifest.config.toolOutput.overflow, + }; + } + if (manifest.config.dynamicModel !== undefined) { return { ...config, diff --git a/packages/eve/src/runtime/subagents/dynamic-agent-config.ts b/packages/eve/src/runtime/subagents/dynamic-agent-config.ts index c1229c2c33..ce80d56b93 100644 --- a/packages/eve/src/runtime/subagents/dynamic-agent-config.ts +++ b/packages/eve/src/runtime/subagents/dynamic-agent-config.ts @@ -7,6 +7,7 @@ import { isDynamicModelDefinition, type AgentLimitsDefinition, type AgentReasoningDefinition, + type AgentToolOutputDefinition, } from "#shared/agent-definition.js"; import type { JsonObject } from "#shared/json.js"; import { serializeOutputSchema } from "#tools/schema.js"; @@ -21,6 +22,7 @@ export interface DynamicSubagentAgentConfig { readonly model: DynamicSubagentModelReference; readonly outputSchema?: JsonObject; readonly reasoning?: AgentReasoningDefinition; + readonly toolOutput?: AgentToolOutputDefinition; } export type DynamicSubagentModelReference = RuntimeModelReference; @@ -54,6 +56,7 @@ export async function normalizeDynamicSubagentAgentConfig(input: { model: DynamicSubagentModelReference; outputSchema?: JsonObject; reasoning?: AgentReasoningDefinition; + toolOutput?: AgentToolOutputDefinition; } = { description: definition.description, model: await normalizeDurableModelSelection({ @@ -97,6 +100,9 @@ export async function normalizeDynamicSubagentAgentConfig(input: { if (definition.reasoning !== undefined) { config.reasoning = definition.reasoning; } + if (definition.toolOutput !== undefined) { + config.toolOutput = definition.toolOutput; + } return config; } diff --git a/packages/eve/src/shared/agent-definition.ts b/packages/eve/src/shared/agent-definition.ts index 1ff61d2454..6214758fe7 100644 --- a/packages/eve/src/shared/agent-definition.ts +++ b/packages/eve/src/shared/agent-definition.ts @@ -140,6 +140,16 @@ export interface PublicAgentCompactionDefinition { readonly thresholdPercent?: number; } +/** + * Controls how eve stores oversized model-facing tool results. + */ +export interface AgentToolOutputDefinition { + /** Maximum UTF-8 payload size that remains inline in model history. */ + readonly maxInlineBytes: number; + /** Writes oversized payloads to the session sandbox and leaves a file reference inline. */ + readonly overflow: "sandbox"; +} + /** * Configures framework-owned runtime limits for this agent's runs. */ @@ -268,6 +278,7 @@ export type InternalAgentDefinition = { reasoning?: AgentReasoningDefinition; source?: ModuleSourceRef; limits?: AgentLimitsDefinition; + toolOutput?: AgentToolOutputDefinition; }; /** @@ -307,6 +318,10 @@ type PublicAgentDefinitionBase = { * per-message output schema. */ readonly outputSchema?: StandardJSONSchemaV1 | JsonObject; + /** + * Optional agent-wide overflow policy for model-facing tool results. + */ + readonly toolOutput?: AgentToolOutputDefinition; }; /** diff --git a/packages/eve/src/tasks/delivery-context.test.ts b/packages/eve/src/tasks/delivery-context.test.ts index 377c98d72e..145f28238f 100644 --- a/packages/eve/src/tasks/delivery-context.test.ts +++ b/packages/eve/src/tasks/delivery-context.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import type { SessionStateMap } from "#harness/types.js"; +import { mockSandbox } from "#internal/testing/mocks/mock-sandbox.js"; import { EMPTY_DELIVERY_SENTINEL } from "#shared/empty-delivery.js"; import { resolveInitiatingTaskContext, @@ -68,9 +69,9 @@ describe("task delivery instructions", () => { }); describe("resolveInitiatingTaskContext", () => { - it("projects the active turn's accepted background tasks as initiating", () => { + it("projects the active turn's accepted background tasks as initiating", async () => { expect( - resolveInitiatingTaskContext({ + await resolveInitiatingTaskContext({ state: taskState([ taskEntry("task_1", "turn_1", undefined, { data: {}, kind: "subagent" }), taskEntry("task_2", "turn_2", undefined, { data: {}, kind: "subagent" }), @@ -81,12 +82,13 @@ describe("resolveInitiatingTaskContext", () => { context: '[Task state]\n{"tasks":[{"name":"report_probe","status":"pending","taskId":"task_1"}]}', phase: "initiating", + spills: [], }); }); - it("ignores task records that were not accepted by an executor", () => { + it("ignores task records that were not accepted by an executor", async () => { expect( - resolveInitiatingTaskContext({ + await resolveInitiatingTaskContext({ state: taskState([taskEntry("task_1", "turn_1")]), turnId: "turn_1", }), @@ -95,7 +97,7 @@ describe("resolveInitiatingTaskContext", () => { }); describe("resolveTaskDeliveryContext", () => { - it("projects terminal and pending siblings from the delivered task's parent turn", () => { + it("projects terminal and pending siblings from the delivered task's parent turn", async () => { const completed = { lastOutput: { data: { result: "first" }, type: "result" }, metadata, @@ -108,16 +110,17 @@ describe("resolveTaskDeliveryContext", () => { taskEntry("task_3", "turn_2"), ]); - expect(resolveTaskDeliveryContext({ state, taskDeliveryId: "task_1:ready:completed" })).toEqual( - { - context: - '[Task state]\n{"tasks":[{"name":"report_probe","status":"completed","taskId":"task_1"},{"name":"report_probe","status":"pending","taskId":"task_2"}]}', - phase: "pending", - }, - ); + expect( + await resolveTaskDeliveryContext({ state, taskDeliveryId: "task_1:ready:completed" }), + ).toEqual({ + context: + '[Task state]\n{"tasks":[{"name":"report_probe","status":"completed","taskId":"task_1"},{"name":"report_probe","status":"pending","taskId":"task_2"}]}', + phase: "pending", + spills: [], + }); }); - it("includes every output once the parent has received the whole terminal cohort", () => { + it("includes every output once the parent has received the whole terminal cohort", async () => { const first = { lastOutput: { data: { result: "first" }, type: "result" }, metadata, @@ -132,7 +135,7 @@ describe("resolveTaskDeliveryContext", () => { } satisfies TaskView; expect( - resolveTaskDeliveryContext({ + await resolveTaskDeliveryContext({ state: taskState([ taskEntry("task_1", "turn_1", first), taskEntry("task_2", "turn_1", second), @@ -143,12 +146,42 @@ describe("resolveTaskDeliveryContext", () => { context: '[Task state]\n{"tasks":[{"name":"report_probe","output":{"data":{"result":"first"},"type":"result"},"status":"completed","taskId":"task_1"},{"name":"report_probe","output":{"data":{"result":"second"},"type":"result"},"status":"completed","taskId":"task_2"}]}', phase: "settled", + spills: [], }); }); - it("returns no context when the delivery is not owned by the session task index", () => { + it("projects terminal output through the configured overflow policy", async () => { + const sandbox = mockSandbox(); + const completed = { + lastOutput: { data: { result: "x".repeat(100) }, type: "result" }, + metadata, + status: "completed", + taskId: "task_1", + } satisfies TaskView; + + const result = await resolveTaskDeliveryContext({ + policy: { maxInlineBytes: 32, overflow: "sandbox" }, + sandboxAccess: sandbox.access, + state: taskState([taskEntry("task_1", "turn_1", completed)]), + taskDeliveryId: "task_1:ready:completed", + }); + + expect(sandbox.writes).toHaveLength(1); + expect(result?.context).toContain('"kind":"eve-tool-output-file"'); + expect(result?.context).not.toContain("x".repeat(100)); + expect(result?.spills).toEqual([ + expect.objectContaining({ + callId: "task:task_1", + maxInlineBytes: 32, + path: sandbox.writes[0]?.path, + toolName: "task", + }), + ]); + }); + + it("returns no context when the delivery is not owned by the session task index", async () => { expect( - resolveTaskDeliveryContext({ + await resolveTaskDeliveryContext({ state: taskState([taskEntry("task_1", "turn_1")]), taskDeliveryId: "task_unknown:ready:completed", }), diff --git a/packages/eve/src/tasks/delivery-context.ts b/packages/eve/src/tasks/delivery-context.ts index 6a6b0866d9..1864be3beb 100644 --- a/packages/eve/src/tasks/delivery-context.ts +++ b/packages/eve/src/tasks/delivery-context.ts @@ -1,4 +1,17 @@ -import type { SessionStateMap } from "#harness/types.js"; +import { ContextContainer } from "#context/container.js"; +import { + PendingToolOutputSpillsKey, + SandboxKey, + TurnTaskDeliveryKey, + TurnTaskStateKey, +} from "#context/keys.js"; +import type { SandboxAccess } from "#sandbox/state.js"; +import type { AgentToolOutputDefinition } from "#shared/agent-definition.js"; +import { + projectOversizedToolOutputValue, + type ToolOutputSpill, +} from "#harness/tool-output-overflow.js"; +import type { SessionStateMap, StepInput } from "#harness/types.js"; import { EMPTY_DELIVERY_SENTINEL } from "#shared/empty-delivery.js"; import { getSessionTaskIndex, type SessionTaskIndexEntry } from "#tasks/session-index.js"; @@ -29,47 +42,146 @@ Correct: ${EMPTY_DELIVERY_SENTINEL}`; export const TASK_DELIVERY_SETTLED_INSTRUCTION = `Background task reporting\nThis turn was triggered by background task activity. The accompanying ${TASK_DELIVERY_CONTEXT_LABEL} message is runtime-authored and lists tasks started by the same parent turn, all settled, with every available terminal output. Do not reply with ${EMPTY_DELIVERY_SENTINEL}. Send one user-facing response that combines their useful results.`; +export async function prepareTaskDeliveryModelContext(input: { + readonly ctx: ContextContainer; + readonly policy?: AgentToolOutputDefinition; + readonly resolved: StepInput | undefined; + readonly state: SessionStateMap | undefined; + readonly taskDeliveryId: string | undefined; + readonly tasksEnabled: boolean; + readonly turnId: string; +}): Promise { + let resolved = input.resolved; + let spills: readonly ToolOutputSpill[] = []; + if (resolved !== undefined && input.taskDeliveryId !== undefined) { + const taskContext = await resolveTaskDeliveryContext({ + policy: input.policy, + sandboxAccess: input.ctx.get(SandboxKey), + state: input.state, + taskDeliveryId: input.taskDeliveryId, + }); + if (taskContext !== undefined) { + input.ctx.set(TurnTaskDeliveryKey, taskContext.phase); + resolved = { ...resolved, context: [...(resolved.context ?? []), taskContext.context] }; + spills = taskContext.spills; + } + } + + if (input.tasksEnabled && input.ctx.get(TurnTaskDeliveryKey) === "none") { + const taskContext = await resolveInitiatingTaskContext({ + policy: input.policy, + sandboxAccess: input.ctx.get(SandboxKey), + state: input.state, + turnId: input.turnId, + }); + if (taskContext !== undefined) { + input.ctx.set(TurnTaskDeliveryKey, taskContext.phase); + input.ctx.set(TurnTaskStateKey, taskContext.context); + spills = taskContext.spills; + } + } + + if (spills.length > 0) input.ctx.set(PendingToolOutputSpillsKey, spills); + return resolved; +} + /** Returns model context and cohort phase for tasks started by the same parent turn as this delivery. */ -export function resolveTaskDeliveryContext(input: { +export async function resolveTaskDeliveryContext(input: { + readonly policy?: AgentToolOutputDefinition; + readonly sandboxAccess?: SandboxAccess; readonly state: SessionStateMap | undefined; readonly taskDeliveryId: string; -}): { readonly context: string; readonly phase: "pending" | "settled" } | undefined { +}): Promise< + | { + readonly context: string; + readonly phase: "pending" | "settled"; + readonly spills: readonly ToolOutputSpill[]; + } + | undefined +> { const entries = getSessionTaskIndex(input.state); const delivered = entries.find((entry) => input.taskDeliveryId.startsWith(`${entry.taskId}:`)); if (delivered === undefined) return undefined; const cohort = entries.filter((entry) => entry.createdByTurnId === delivered.createdByTurnId); - return projectTaskCohort(cohort); + return projectTaskCohort({ + cohort, + policy: input.policy, + sandboxAccess: input.sandboxAccess, + }); } /** Returns model context for durable tasks launched by the active parent turn. */ -export function resolveInitiatingTaskContext(input: { +export async function resolveInitiatingTaskContext(input: { + readonly policy?: AgentToolOutputDefinition; + readonly sandboxAccess?: SandboxAccess; readonly state: SessionStateMap | undefined; readonly turnId: string; -}): { readonly context: string; readonly phase: "initiating" } | undefined { +}): Promise< + | { + readonly context: string; + readonly phase: "initiating"; + readonly spills: readonly ToolOutputSpill[]; + } + | undefined +> { const cohort = getSessionTaskIndex(input.state).filter( (entry) => entry.createdByTurnId === input.turnId, ); if (!cohort.some((entry) => entry.executor !== undefined && entry.terminalView === undefined)) { return undefined; } - return { ...projectTaskCohort(cohort), phase: "initiating" }; + return { + ...(await projectTaskCohort({ + cohort, + policy: input.policy, + sandboxAccess: input.sandboxAccess, + })), + phase: "initiating", + }; } -function projectTaskCohort(cohort: readonly SessionTaskIndexEntry[]): { +async function projectTaskCohort(input: { + readonly cohort: readonly SessionTaskIndexEntry[]; + readonly policy: AgentToolOutputDefinition | undefined; + readonly sandboxAccess: SandboxAccess | undefined; +}): Promise<{ readonly context: string; readonly phase: "pending" | "settled"; -} { - const settled = cohort.every((entry) => entry.terminalView !== undefined); - const tasks = cohort.map((entry) => ({ - name: entry.metadata.name, - output: settled ? entry.terminalView?.lastOutput : undefined, - status: entry.terminalView?.status ?? "pending", - taskId: entry.taskId, - })); + readonly spills: readonly ToolOutputSpill[]; +}> { + const settled = input.cohort.every((entry) => entry.terminalView !== undefined); + const spills: ToolOutputSpill[] = []; + const tasks = await Promise.all( + input.cohort.map(async (entry) => { + const lastOutput = settled ? entry.terminalView?.lastOutput : undefined; + const output = + lastOutput === undefined + ? undefined + : { + ...lastOutput, + data: await projectOversizedToolOutputValue({ + output: lastOutput.data, + onSpill: (spill) => { + spills.push(spill); + }, + policy: input.policy, + sandboxAccess: input.sandboxAccess, + taskId: entry.taskId, + }), + }; + return { + name: entry.metadata.name, + output, + status: entry.terminalView?.status ?? "pending", + taskId: entry.taskId, + }; + }), + ); return { context: `${TASK_DELIVERY_CONTEXT_LABEL}\n${JSON.stringify({ tasks })}`, phase: settled ? "settled" : "pending", + spills, }; } diff --git a/packages/eve/src/tracing/agent-otel-provider.test.ts b/packages/eve/src/tracing/agent-otel-provider.test.ts index 4adae02eb0..15c8aac9b9 100644 --- a/packages/eve/src/tracing/agent-otel-provider.test.ts +++ b/packages/eve/src/tracing/agent-otel-provider.test.ts @@ -44,6 +44,7 @@ import { modelCallIdempotencyKey, sessionIdempotencyKey, turnIdempotencyKey, + toolOutputSpillIdempotencyKey, } from "#harness/instrumentation/lifecycle.js"; interface TestRuntime { @@ -521,6 +522,49 @@ describe("createAgentOtelInstrumentation", () => { expect(byName(spans, "agent.session")).toHaveLength(0); }); + it("records bounded metadata for model-facing tool-output spills", async () => { + const runtime = createRuntime(); + await publishTurnStarted({ + hooks: runtime.hooks, + sessionId: "session-1", + turnId: "turn-1", + turnSequence: 0, + }); + await runtime.hooks.publish({ + bytes: 131_072, + callId: "call-1", + idempotencyKey: toolOutputSpillIdempotencyKey("session-1", "abc"), + maxInlineBytes: 65_536, + path: "/workspace/.eve/tool-results/abc.json", + sequence: 0, + sessionId: "session-1", + spillId: "abc", + stepIndex: 1, + toolName: "search", + turnId: "turn-1", + type: "tool.output.spilled", + }); + await completeTurn(runtime.hooks, "session-1", "turn-1"); + await runtime.provider.forceFlush(); + + const spans = runtime.exporter.getFinishedSpans(); + const spill = byName(spans, "agent.tool.output")[0]!; + const turn = byName(spans, "agent.turn")[0]!; + expect(spill.parentSpanContext?.spanId).toBe(turn.spanContext().spanId); + expect(spill.events.map((event) => event.name)).toEqual(["tool.output.spilled"]); + expect(spill.attributes).toMatchObject({ + "agent.session.id": "session-1", + "agent.step.index": 1, + "agent.tool.output.bytes": 131_072, + "agent.tool.output.call_id": "call-1", + "agent.tool.output.max_inline_bytes": 65_536, + "agent.tool.output.path": "/workspace/.eve/tool-results/abc.json", + "agent.tool.output.spill_id": "abc", + "agent.tool.output.tool_name": "search", + "agent.turn.id": "turn-1", + }); + }); + it.each([ ["cancelled", SpanStatusCode.UNSET], ["failed", SpanStatusCode.ERROR], diff --git a/packages/eve/src/tracing/agent-otel-provider.ts b/packages/eve/src/tracing/agent-otel-provider.ts index a618e9fe97..820ffca9d5 100644 --- a/packages/eve/src/tracing/agent-otel-provider.ts +++ b/packages/eve/src/tracing/agent-otel-provider.ts @@ -42,6 +42,7 @@ import type { InstrumentationProviderDefinition, InstrumentationSessionStartedEvent, InstrumentationTraceContext, + InstrumentationToolOutputSpilledEvent, InstrumentationSessionTransitionEvent, InstrumentationTurnStartedEvent, InstrumentationTurnTerminalEvent, @@ -428,6 +429,40 @@ export function createAgentOtelInstrumentation( } }; + const onToolOutputSpilled = async ( + event: InstrumentationToolOutputSpilledEvent, + ): Promise => { + const turn = await input.stateStore.getTurn(event.sessionId, event.turnId); + const parent = turn?.context ?? (await input.stateStore.getSession(event.sessionId))?.context; + if (parent === undefined || !isSampledTrace(parent)) return; + + const span = input.idGenerator.withSpanId( + input.idGenerator.deriveSpanId(event.idempotencyKey), + () => + input.tracer.startSpan( + "agent.tool.output", + { + attributes: { + "agent.framework.name": "eve", + "agent.framework.version": input.frameworkVersion, + "agent.session.id": event.sessionId, + "agent.step.index": event.stepIndex, + "agent.tool.output.bytes": event.bytes, + "agent.tool.output.call_id": event.callId, + "agent.tool.output.max_inline_bytes": event.maxInlineBytes, + "agent.tool.output.path": event.path, + "agent.tool.output.spill_id": event.spillId, + "agent.tool.output.tool_name": event.toolName, + "agent.turn.id": event.turnId, + }, + }, + contextFromSpanContext(parent), + ), + ); + span.addEvent("tool.output.spilled"); + span.end(); + }; + return { hook: { // The destinations behind this pipeline filter content per exporter, but @@ -455,6 +490,7 @@ export function createAgentOtelInstrumentation( "session.started": onSessionStarted, "session.waiting": onSessionTransition, ...tools.events, + "tool.output.spilled": onToolOutputSpilled, "turn.cancelled": onTurnTerminal, "turn.completed": onTurnTerminal, "turn.failed": onTurnTerminal, diff --git a/research/tool-output-overflow.md b/research/tool-output-overflow.md new file mode 100644 index 0000000000..7b1c7acc3e --- /dev/null +++ b/research/tool-output-overflow.md @@ -0,0 +1,97 @@ +--- +issue: https://github.com/vercel/eve/issues/905 +status: implemented +last_updated: "2026-08-25" +--- + +# Tool output overflow + +## Summary + +A single tool result can consume most of a model context window before +compaction can act. Authored tools can reduce their own output with +`toModelOutput`, but generated connection tools and other runtime-owned paths +do not share an author-controlled projection hook. + +Add one opt-in agent policy that moves oversized model-facing tool results to +the session sandbox. Keep the full execution result on `action.result`; store +only the bounded file reference in model history. + +## Authoring API + +```ts +import { defineAgent } from "eve"; + +export default defineAgent({ + model: "anthropic/claude-sonnet-5", + toolOutput: { + maxInlineBytes: 64 * 1024, + overflow: "sandbox", + }, +}); +``` + +Both fields are required. Omitting `toolOutput` preserves existing behavior. +The only supported overflow strategy is `"sandbox"`. + +## Semantics + +The policy runs at the common model-history boundary after runtime action +resolution and `action.result` emission have preserved the full output, but +before compaction and the next model request. + +For each eve-controlled `tool-result`: + +- text is measured in UTF-8 bytes; +- JSON is measured using its compact serialization; +- results at or below `maxInlineBytes` remain unchanged; +- larger text and JSON values are written as `.txt` and readable `.json` files + under `/workspace/.eve/tool-results`; +- model history receives `{ kind, path, bytes, toolName }` as a JSON tool + result, tagged with `kind: "eve-tool-output-file"`. + +The filename derives deterministically from the tool call id and serialized +output. Replaying the same call with the same output reuses its path; a changed +output receives a different path. Existing eve file references are not +projected again. + +After a sandbox write succeeds, eve emits `tool.output.spilled` with the call +and tool identity, byte count, configured limit, path, and a deterministic +`spillId` derived from the same digest as the file path. The event is durable +and available to hooks, channels, and stream clients. Instrumentation projects +the same bounded metadata onto an `agent.tool.output` span without recording +the full body. + +Within a committed session, the stored reference prevents a later projection +from emitting again. A durable-step retry can still physically repeat the +event because event-stream writes and the returned session checkpoint are not +one transaction. Consumers and trace backends use `spillId` as the logical +idempotency key. + +Framework control results that later steps must parse from history remain +inline. `connection_search` is the initial protected control tool because its +results reconstruct discovered connection tools. Approval denials also remain +inline. + +## Boundaries + +The policy covers authored tools, framework tools, discovered connection +tools, client-supplied tools, runtime-action results, and subagent or task +results that eve places in model history or runtime-authored task context. +Explicit `toModelOutput` projections run first and remain the preferred way to +provide a semantic summary. + +Provider-executed server tools are outside the first-call boundary because the +provider consumes their output before eve receives it. Sandbox storage is +session-scoped, not external artifact storage; provider-side physical sandbox +replacement may lose files created after sandbox initialization. + +## Verification + +- Public config normalization and compiled-manifest propagation. +- Small-result passthrough and oversized text/JSON projection. +- Discovered MCP-shaped output, deterministic replay paths, protected control + output, and reference idempotence. +- Full `action.result` emission, durable `tool.output.spilled` notification, + bounded trace metadata, a referenced first checkpoint, and the same reference + on the next model call.