diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3d3388e0..a128d19f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -80,6 +80,9 @@ jobs: - name: Test recovery MCP product run: pnpm --dir components/products/mcp test + - name: Test OpenCode adapter + run: pnpm --dir components/adapters/opencode test + - name: Test lightrsi CLI run: pnpm --dir components/products/cli test diff --git a/components/adapters/HOSTS.md b/components/adapters/HOSTS.md index 9ecb25c1..ad2f9cce 100644 --- a/components/adapters/HOSTS.md +++ b/components/adapters/HOSTS.md @@ -10,6 +10,7 @@ TokenPilot is now structured as a reusable LightRSI component with multiple host | `Codex CLI` | available | hooks + local Responses proxy + shared CLI | `npm --prefix components/adapters/codex run build` then `npm --prefix components/adapters/codex run install:codex` | [codex/README.md](./codex/README.md) | | `Claude Code` | available | gateway routing + observability hooks + shared CLI | `npm --prefix components/adapters/claude-code run build` then `npm --prefix components/adapters/claude-code run install:claude-code` | [claude-code/README.md](./claude-code/README.md) | | `DeepSeek Harness` | available | Cordis plugin + durable projection | `pnpm --filter @lightrsi/deepseek-harness-adapter compatibility:smoke -- --dsh-checkout=/absolute/path/to/deepseek-harness` | [GUA-06](../../docs/acceptance/GUA-06.md) | +| `OpenCode` | available | in-process v1 plugin + recovery MCP + shared CLI | `npm --prefix components/adapters/opencode run build` then `npm --prefix components/adapters/opencode run install:opencode` | [opencode/README.md](./opencode/README.md) | Each full TokenPilot host adapter binds the versioned preset explicitly. OpenClaw and Codex declare Stabilizer, Reduction, and Eviction; Claude Code currently declares Stabilizer and Reduction. DeepSeek Harness is a separate compatibility adapter with a narrower projection surface. The same adapter-owned product registrations are used by the shared CLI and browser Visual surface where applicable. @@ -21,22 +22,22 @@ Legend: - `partial`: available, but intentionally narrower than the OpenClaw path - `no`: not supported in the current public adapter -| Capability | OpenClaw | Codex CLI | Claude Code | -| :-- | :--: | :--: | :--: | -| Stable-prefix rewriting | yes | yes | yes | -| Before-call reduction | yes | yes | yes | -| Real MCP-backed `memory_fault_recover` | yes | yes | yes | -| Standalone `lightrsi ...` CLI | yes | yes | yes | -| `status` / `doctor` / `report` | yes | yes | yes | -| `visual` | yes | yes | yes | -| `mode conservative` / `mode normal` | yes | yes | yes | -| `mode aggressive` | yes | no | no | -| Estimator-driven lifecycle eviction runtime | yes | yes | yes | -| Lifecycle eviction controls | yes | no | no | -| In-host slash commands | yes | no | no | -| Hook-based observability | partial | yes | yes | -| Local proxy / gateway runtime | yes | yes | yes | -| Session-state / ux-effects persistence | yes | yes | yes | +| Capability | OpenClaw | Codex CLI | Claude Code | OpenCode | +| :-- | :--: | :--: | :--: | :--: | +| Stable-prefix rewriting | yes | yes | yes | partial (`developer` target only) | +| Before-call reduction | yes | yes | yes | yes | +| Real MCP-backed `memory_fault_recover` | yes | yes | yes | yes | +| Standalone `lightrsi ...` CLI | yes | yes | yes | yes | +| `status` / `doctor` / `report` | yes | yes | yes | yes | +| `visual` | yes | yes | yes | yes | +| `mode conservative` / `mode normal` | yes | yes | yes | yes | +| `mode aggressive` | yes | no | no | no | +| Estimator-driven lifecycle eviction runtime | yes | yes | yes | yes (durable request overlay) | +| Lifecycle eviction controls | yes | no | no | `on` / `off` / `minBlockChars` | +| In-host slash commands | yes | no | no | no | +| Hook-based observability | partial | yes | yes | in-process | +| Local proxy / gateway runtime | yes | yes | yes | not needed (in-process) | +| Session-state / ux-effects persistence | yes | yes | yes | yes | ## Host Notes @@ -69,10 +70,18 @@ Legend: - supports TokenPilot status projection and compatibility smoke verification - canonical surface eviction remains opt-in and is guarded by estimator, registry, safety, and revision checks +### OpenCode + +- in-process v1 plugin (verified on OpenCode 1.18.33) plus the shared recovery MCP server in `opencode.json` +- stable prefix in `experimental.chat.system.transform`; reduction and opt-in eviction (durable request overlay) in `experimental.chat.messages.transform`, which runs before every model step +- `dynamicContextTarget=user` is not available (messages are transformed before the system prompt in 1.18.33) +- never changes OpenCode's native `compaction.prune`; `doctor` reports it +- design note: [docs/adapters/opencode-design.md](../../docs/adapters/opencode-design.md) + ### Shared Visual Surface - `lightrsi visual` now provides a standalone browser visual entrypoint -- the shared visual can switch between `openclaw`, `codex`, and `claude-code` hosts +- the shared visual can switch between `openclaw`, `codex`, `claude-code`, and `opencode` hosts - today, the browser visual is backed by snapshot data; OpenClaw still has the richest dataset, while Codex and Claude Code now route their `visual` commands into the shared browser surface ## Boundary diff --git a/components/adapters/README.md b/components/adapters/README.md index eef88e48..cca8e5bf 100644 --- a/components/adapters/README.md +++ b/components/adapters/README.md @@ -23,6 +23,10 @@ Adapter inventory: - adapter for Claude Code - `deepseek-harness/` - Cordis adapter and compatibility smoke path for DeepSeek Harness +- `opencode/` + - in-process v1 plugin adapter for OpenCode +- `shared/canonical/` + - host glue for in-process transcript adapters (canonical reduction, stabilizer application, recovery protocol, eviction surface, fail-open logging) - future adapters - other host-specific integrations @@ -126,3 +130,4 @@ This keeps the first working version small and makes boundary mistakes easier to - [codex/README.md](./codex/README.md) - [claude-code/README.md](./claude-code/README.md) - [deepseek-harness/](./deepseek-harness/) +- [opencode/README.md](./opencode/README.md) diff --git a/components/adapters/opencode/README.md b/components/adapters/opencode/README.md new file mode 100644 index 00000000..45ce770b --- /dev/null +++ b/components/adapters/opencode/README.md @@ -0,0 +1,134 @@ +# TokenPilot OpenCode Adapter + +This package integrates TokenPilot with [OpenCode](https://opencode.ai) (verified on **1.18.33**) as an in-process v1 plugin, plus the shared recovery MCP server registered in OpenCode's own config. + +This adapter explicitly binds the TokenPilot `stabilizer`, `reduction`, and `eviction` features. Its product registration provides OpenCode state discovery to the shared CLI and Visual surface. + +Design, host facts and gaps: [`docs/adapters/opencode-design.md`](../../../docs/adapters/opencode-design.md). + +## Supports + +| Feature | OpenCode hook | Notes | +| :-- | :-- | :-- | +| Stable prefix | `experimental.chat.system.transform` | Shared stabilizer: volatile lines (e.g. `Today's date: …`) move to the tail of the system prompt. | +| Reduction | `experimental.chat.messages.transform` | Runs before every model step (verified in 1.18.33). Request-time, over the whole history. Stored message parts keep the originals. | +| Recovery | shared recovery MCP server | Registered as `mcp.tokenpilot_memory_fault_recover`. OpenCode shows the tool as `tokenpilot_memory_fault_recover_memory_fault_recover`. The recovery protocol is appended to the system prompt. | +| Eviction (opt-in, off by default) | `experimental.chat.messages.transform` + durable overlay | OpenCode has no persistent part-edit hook for plugins, so evictions are stored per session and re-applied identically on every request (Claude Code's overlay approach). Estimator failures never disable reduction. | +| Modes | | `conservative`, `normal` (default). `aggressive` is not exposed. | + +Current limitations: + +- `hooks.dynamicContextTarget=user` runs as `developer`: in 1.18.33, `messages.transform` runs before `system.transform`, so volatile lines cannot be moved into the same request's first user message. `doctor` reports this. +- the v2 plugin API is not used: in 1.18.33 it has no message, tool or session hooks +- no in-host slash commands; use the shared `lightrsi opencode ...` CLI +- the Context Cleaner is not integrated yet (see the design note) + +### Native tool-output pruning + +OpenCode can prune old tool outputs itself (`compaction.prune`: keeps roughly the last 40k tokens of tool output and clears older ones to `[Old tool result content cleared]`). It is **off by default in 1.18.33**. TokenPilot never changes this setting. `lightrsi opencode doctor` reports it as `native tool-output pruning: on|off`. + +- Off: TokenPilot reduction is the only thing trimming old outputs, apart from OpenCode's own truncation at 2000 lines / 50 KiB (`tool_output.max_lines|max_bytes`). +- On: pruned outputs are skipped by TokenPilot, since they are below the reduction thresholds and are never evicted. Pruning rewrites earlier history persistently, which moves the cached-prefix boundary; that trade-off is OpenCode's and yours to choose. + +## Install + +```bash +pnpm install +pnpm --filter @lightrsi/mcp build # recovery MCP server +npm --prefix components/adapters/opencode run build +npm --prefix components/products/cli run build # for the lightrsi CLI +npm --prefix components/adapters/opencode run install:opencode +``` + +Install writes, in OpenCode's global config dir (`$XDG_CONFIG_HOME/opencode`, default `~/.config/opencode`): + +- `plugins/tokenpilot.js`: a marker-tagged ESM loader re-exporting `components/adapters/opencode/dist/plugin.mjs`. OpenCode auto-loads `plugins/*.js`. A foreign file at that path is moved to `tokenpilot.js.bak-` first. +- `opencode.json`: adds exactly one key, `mcp.tokenpilot_memory_fault_recover`, after copying the file to `opencode.json.tokenpilot-backup-`. The file is created if it does not exist. +- `tokenpilot.json`: runtime config in `normal` mode, created only if missing and never containing credentials. +- `tokenpilot-install.json`: install manifest used by uninstall. + +If you only have an `opencode.jsonc`, or your `opencode.json` is not plain JSON, install **does not rewrite it**. It prints the MCP snippet for you to paste, so your comments are never lost. `doctor` flags the missing entry. + +Uninstall removes the loader and exactly that MCP key (and an `opencode.json` it created, once nothing else is in it): + +```bash +npm --prefix components/adapters/opencode run uninstall:opencode +npm --prefix components/adapters/opencode run uninstall:opencode -- --purge # also created config + state +``` + +`TOKENPILOT_OPENCODE_CONFIG_DIR` and `TOKENPILOT_OPENCODE_CONFIG` override the locations. + +## Verify + +1. Restart OpenCode so the plugin and the MCP server load. +2. `lightrsi opencode doctor`: the loader, bundle and MCP entry should be present, and the MCP probe should be `ok`. +3. After a few turns: `lightrsi opencode report` + +## Commands + +```bash +lightrsi opencode status +lightrsi opencode report +lightrsi opencode doctor +lightrsi opencode visual +lightrsi opencode mode conservative|normal +lightrsi opencode stabilizer on|off +lightrsi opencode reduction on|off +lightrsi opencode reduction pass on|off +lightrsi opencode eviction status|on|off +lightrsi opencode eviction set minBlockChars +``` + +## Configuration + +`~/.config/opencode/tokenpilot.json` uses the same keys and defaults as the Claude Code adapter. Config edits, including `lightrsi opencode mode ...`, apply on the next request without restarting OpenCode. + +| Key | Default | Notes | +| :-- | :-- | :-- | +| `enabled` | `true` | Master switch; `false` makes every hook a no-op. | +| `modules.stabilizer` | `true` | | +| `hooks.dynamicContextTarget` | `developer` | `user` is not available on OpenCode 1.18.33 (see Supports above); it runs as `developer`. | +| `modules.reduction` | `true` | | +| `reduction.triggerMinChars` / `maxToolChars` | `2200` / `1200` | `conservative`: `4000` / `1800`. | +| `reduction.passes.*` | all `true` | | +| `modules.eviction` + `eviction.enabled` | `false` | Both must be on, plus the estimator settings below. | +| `eviction.minBlockChars` | `4000` | | +| `taskStateEstimator.baseUrl` / `model` / `apiKey` | unset | OpenAI-compatible endpoint for the task-state estimator. Keep the key out of checked-in files. | + +Enabling eviction: + +```bash +lightrsi opencode eviction on +# then set taskStateEstimator.baseUrl / model / apiKey in ~/.config/opencode/tokenpilot.json +lightrsi opencode doctor # "eviction: active" once complete +``` + +## Runtime Files + +```text +~/.config/opencode/tokenpilot-state/tokenpilot/ +``` + +- `tokenpilot/adapter.log`: fail-open reasons. OpenCode does not catch plugin errors, so every hook catches its own and continues unmodified. +- `tokenpilot/tool-result-archives//`: recovery archives +- `tokenpilot/reduction-memo/.json`: which read disclosed which file, so a restarted OpenCode (`opencode run --continue`) still honours deliberate re-reads +- `tokenpilot/eviction-overlay/.json`: durable eviction decisions (when eviction is on) +- `session-state/…`, `ux-effects/…`: report and visual data + +## Debugging + +- `lightrsi opencode doctor` +- `tail ~/.config/opencode/tokenpilot-state/tokenpilot/tokenpilot/adapter.log` +- `opencode run --print-logs --log-level DEBUG "…"` shows plugin and MCP loading. +- `opencode run --pure "…"` runs without external plugins, to compare with TokenPilot off. + +## Package Scripts + +```bash +npm --prefix components/adapters/opencode run build +npm --prefix components/adapters/opencode run typecheck +npm --prefix components/adapters/opencode test +npm --prefix components/adapters/opencode run install:opencode +npm --prefix components/adapters/opencode run uninstall:opencode +npm --prefix components/adapters/opencode run doctor:opencode +``` diff --git a/components/adapters/opencode/build.ts b/components/adapters/opencode/build.ts new file mode 100644 index 00000000..d9fbaa97 --- /dev/null +++ b/components/adapters/opencode/build.ts @@ -0,0 +1,51 @@ +import { build } from "esbuild"; +import { join } from "node:path"; + +const shared = { + bundle: true, + // ../shared/canonical has no node_modules of its own; resolve workspace deps from here. + nodePaths: [join(__dirname, "node_modules")], + platform: "node" as const, + target: "node20", + sourcemap: true, + minify: false, + logLevel: "info" as const, + logOverride: { "empty-import-meta": "silent" as const }, +}; + +async function main() { + // OpenCode (Bun) imports plugins as ES modules. Bundled CommonJS dependencies + // still need `require`, `__filename` and `__dirname`, so provide them. + await build({ + ...shared, + entryPoints: { plugin: "src/plugin.ts" }, + outdir: "dist", + format: "esm", + outExtension: { ".js": ".mjs" }, + banner: { + js: [ + 'import { createRequire as __tpCreateRequire } from "node:module";', + 'import { fileURLToPath as __tpFileURLToPath } from "node:url";', + 'import { dirname as __tpDirname } from "node:path";', + "const require = __tpCreateRequire(import.meta.url);", + "const __filename = __tpFileURLToPath(import.meta.url);", + "const __dirname = __tpDirname(__filename);", + ].join("\n"), + }, + }); + await build({ + ...shared, + entryPoints: { + "install-opencode": "scripts/install-opencode.ts", + "uninstall-opencode": "scripts/uninstall-opencode.ts", + "doctor-opencode": "scripts/doctor-opencode.ts", + }, + outdir: "dist", + format: "cjs", + }); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/components/adapters/opencode/package.json b/components/adapters/opencode/package.json new file mode 100644 index 00000000..fa068cb6 --- /dev/null +++ b/components/adapters/opencode/package.json @@ -0,0 +1,44 @@ +{ + "name": "@lightrsi/opencode-adapter", + "version": "0.1.0-beta.1", + "description": "TokenPilot adapter for OpenCode (in-process v1 plugin + recovery MCP).", + "lightrsi": { + "role": "adapter" + }, + "main": "dist/plugin.js", + "license": "MIT", + "repository": { + "type": "git", + "url": "https://github.com/zjunlp/LightRSI.git", + "directory": "components/adapters/opencode" + }, + "engines": { + "node": ">=20.0.0" + }, + "dependencies": { + "@lightrsi/artifact-store": "workspace:*", + "@lightrsi/eviction": "workspace:*", + "@lightrsi/history": "workspace:*", + "@lightrsi/host-adapter": "workspace:*", + "@lightrsi/kernel": "workspace:*", + "@lightrsi/mcp": "workspace:*", + "@lightrsi/product-surface": "workspace:*", + "@lightrsi/reduction": "workspace:*", + "@lightrsi/stabilizer": "workspace:*", + "@lightrsi/tokenpilot": "workspace:*" + }, + "scripts": { + "build": "tsx build.ts", + "doctor:opencode": "node --import tsx scripts/doctor-opencode.ts", + "install:opencode": "node --import tsx scripts/install-opencode.ts", + "uninstall:opencode": "node --import tsx scripts/uninstall-opencode.ts", + "test": "node --import tsx --test tests/*.test.ts", + "typecheck": "tsc -p tsconfig.json --noEmit" + }, + "devDependencies": { + "esbuild": "^0.25.10", + "tsx": "^4.21.0", + "typescript": "^5.9.3" + }, + "keywords": ["opencode", "tokenpilot", "context-management", "plugin"] +} diff --git a/components/adapters/opencode/scripts/doctor-opencode.ts b/components/adapters/opencode/scripts/doctor-opencode.ts new file mode 100644 index 00000000..1b307146 --- /dev/null +++ b/components/adapters/opencode/scripts/doctor-opencode.ts @@ -0,0 +1,16 @@ +#!/usr/bin/env node +import { defaultTokenPilotOpenCodeConfigPath, loadTokenPilotOpenCodeConfig } from "../src/config.js"; +import { formatOpenCodeDoctorReport, inspectOpenCodeDoctor } from "../src/doctor.js"; + +async function main() { + const configPath = process.env.TOKENPILOT_OPENCODE_CONFIG ?? defaultTokenPilotOpenCodeConfigPath(); + const config = await loadTokenPilotOpenCodeConfig(configPath); + const report = await inspectOpenCodeDoctor({ config, configPath, configDir: process.env.TOKENPILOT_OPENCODE_CONFIG_DIR }); + console.log(formatOpenCodeDoctorReport(report)); + if (!report.healthy) process.exitCode = 1; +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); +}); diff --git a/components/adapters/opencode/scripts/install-opencode.ts b/components/adapters/opencode/scripts/install-opencode.ts new file mode 100644 index 00000000..a23fa249 --- /dev/null +++ b/components/adapters/opencode/scripts/install-opencode.ts @@ -0,0 +1,33 @@ +#!/usr/bin/env node +import { installOpenCodeTokenPilot } from "../src/install.js"; + +async function main() { + const result = await installOpenCodeTokenPilot({ + configDir: process.env.TOKENPILOT_OPENCODE_CONFIG_DIR, + tokenPilotConfigPath: process.env.TOKENPILOT_OPENCODE_CONFIG, + }); + const lines = [ + "TokenPilot OpenCode install complete:", + `- plugin loader: ${result.paths.loaderPath}`, + `- plugin bundle: ${result.bundlePath}`, + ...(result.loaderBackupPath ? [`- previous loader backed up to: ${result.loaderBackupPath}`] : []), + `- tokenpilot config: ${result.paths.tokenPilotConfigPath} (${result.tokenPilotConfigCreated ? "created, normal mode" : "kept existing"})`, + `- state dir: ${result.stateDir}`, + `- recovery MCP: ${result.mcp.registered ? `${result.mcp.reason} in ${result.paths.opencodeJsonPath}` : `NOT registered (${result.mcp.reason})`}`, + ...(result.mcp.backupPath ? [`- opencode.json backup: ${result.mcp.backupPath}`] : []), + ...(!result.mcp.serverBuilt ? ["- recovery MCP server is not built yet: run `pnpm --filter @lightrsi/mcp build`"] : []), + ...(!result.mcp.registered ? ["- add this to your OpenCode config manually:", result.mcp.snippet] : []), + result.cliBin?.installed + ? `- lightrsi CLI bin: installed at ${result.cliBin.launcherPath ?? result.cliBin.binPath}` + : "- lightrsi CLI bin: skipped (build components/products/cli first)", + ...(result.cliBin && !result.cliBin.binDirOnPath ? [`- lightrsi CLI PATH note: add ${result.cliBin.binDir} to PATH if 'lightrsi' is unavailable.`] : []), + "- next step: restart OpenCode so the plugin and MCP server load", + "- verify: lightrsi opencode doctor", + ]; + console.log(lines.join("\n")); +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); +}); diff --git a/components/adapters/opencode/scripts/uninstall-opencode.ts b/components/adapters/opencode/scripts/uninstall-opencode.ts new file mode 100644 index 00000000..e3873948 --- /dev/null +++ b/components/adapters/opencode/scripts/uninstall-opencode.ts @@ -0,0 +1,26 @@ +#!/usr/bin/env node +import { uninstallOpenCodeTokenPilot } from "../src/install.js"; + +async function main() { + const purge = process.argv.includes("--purge"); + const result = await uninstallOpenCodeTokenPilot({ + configDir: process.env.TOKENPILOT_OPENCODE_CONFIG_DIR, + tokenPilotConfigPath: process.env.TOKENPILOT_OPENCODE_CONFIG, + purge, + }); + console.log([ + "TokenPilot OpenCode uninstall:", + `- plugin loader removed: ${result.loaderRemoved ? "yes" : "no"}`, + `- previous loader restored: ${result.loaderBackupRestored ? "yes" : "no"}`, + ...(result.foreignLoaderKept ? ["- a non-TokenPilot plugin exists at the path and was left untouched"] : []), + `- recovery MCP entry removed: ${result.mcpRemoved ? "yes" : "no"}`, + ...(result.opencodeConfigRemoved ? ["- opencode.json created by install was removed (it held nothing else)"] : []), + `- config removed: ${result.configRemoved ? "yes" : purge ? "no (not created by install)" : "no (pass --purge)"}`, + `- state removed: ${result.stateRemoved ? "yes" : purge ? "no" : "no (pass --purge)"}`, + ].join("\n")); +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); +}); diff --git a/components/adapters/opencode/src/codec.ts b/components/adapters/opencode/src/codec.ts new file mode 100644 index 00000000..b97e27db --- /dev/null +++ b/components/adapters/opencode/src/codec.ts @@ -0,0 +1,207 @@ +/** + * OpenCode transcript bridge: `{ info, parts }[]` ↔ canonical `RuntimeMessage[]`. + * + * Mirrors what OpenCode 1.18.33 sends the model (`session/message-v2.ts` + * `toModelMessagesEffect`): + * user text (not `ignored`, not empty) → user text blocks; image files → image blocks + * assistant text → text; reasoning → metadata; completed/error tool part → + * tool_call on the assistant message + a following canonical `tool` message + * with the output (the `[Old tool result content cleared]` placeholder when + * natively pruned, the error text for errors) + * pending/running tool parts, step/snapshot/patch/agent/retry/compaction/subtask + * parts → not decoded (never rewritten) + * + * Every canonical message remembers where it came from (`OcBinding`). Encoding + * writes changed text into CLONED parts/messages and swaps them into the array; + * stored objects are never mutated, and an unchanged envelope leaves the array + * untouched. + */ +import type { RuntimeContentBlock, RuntimeMessage } from "@lightrsi/kernel"; +import type { CanonicalSurfaceEntry } from "../../shared/canonical/eviction.js"; +import type { OcMessageWithParts, OcPart, OcTextPart, OcToolPart } from "./opencode-types.js"; + +export const OC_PRUNED_PLACEHOLDER = "[Old tool result content cleared]"; + +export type OcBinding = + | { kind: "user"; messageIndex: number; textPartIndexes: number[] } + | { kind: "assistant"; messageIndex: number; textPartIndexes: number[] } + | { kind: "tool"; messageIndex: number; partIndex: number }; + +export type OcDecoded = { + messages: RuntimeMessage[]; + bindings: OcBinding[]; +}; + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function isToolPart(part: OcPart): part is OcToolPart { + return part?.type === "tool" && isRecord((part as OcToolPart).state); +} + +function toolResultText(part: OcToolPart): { text: string; status: "success" | "error" } | undefined { + const state = part.state; + if (state.status === "completed") { + return { text: state.time?.compacted ? OC_PRUNED_PLACEHOLDER : String(state.output ?? ""), status: "success" }; + } + if (state.status === "error") { + const interrupted = isRecord(state.metadata) && state.metadata.interrupted === true && typeof state.metadata.output === "string"; + return interrupted + ? { text: String((state.metadata as Record).output), status: "success" } + : { text: String(state.error ?? ""), status: "error" }; + } + return undefined; +} + +export function decodeOpenCodeMessages(messages: readonly OcMessageWithParts[]): OcDecoded { + const out: RuntimeMessage[] = []; + const bindings: OcBinding[] = []; + messages.forEach((message, messageIndex) => { + if (!isRecord(message) || !isRecord(message.info) || !Array.isArray(message.parts)) return; + const role = message.info.role; + const meta = { ocMessageId: String(message.info.id ?? ""), ocIndex: messageIndex }; + if (role === "user") { + const blocks: RuntimeContentBlock[] = []; + const textPartIndexes: number[] = []; + message.parts.forEach((part, partIndex) => { + if (!isRecord(part)) return; + if (part.type === "text" && !(part as OcTextPart).ignored && (part as OcTextPart).text !== "") { + blocks.push({ type: "text", text: String((part as OcTextPart).text) }); + textPartIndexes.push(partIndex); + } else if (part.type === "file" && typeof part.mime === "string" && part.mime.startsWith("image/")) { + blocks.push({ type: "image", mediaType: part.mime }); + } + }); + if (blocks.length === 0) return; + out.push({ role: "user", content: blocks, metadata: meta }); + bindings.push({ kind: "user", messageIndex, textPartIndexes }); + return; + } + if (role !== "assistant") return; + const blocks: RuntimeContentBlock[] = []; + const textPartIndexes: number[] = []; + const reasoning: string[] = []; + const results: Array<{ message: RuntimeMessage; binding: OcBinding }> = []; + message.parts.forEach((part, partIndex) => { + if (!isRecord(part)) return; + if (part.type === "text") { + blocks.push({ type: "text", text: String((part as OcTextPart).text ?? "") }); + textPartIndexes.push(partIndex); + } else if (part.type === "reasoning") { + reasoning.push(String(part.text ?? "")); + } else if (isToolPart(part)) { + const result = toolResultText(part); + if (!result) return; + blocks.push({ type: "tool_call", toolCallId: part.callID, toolName: part.tool, argumentsJson: isRecord(part.state.input) ? part.state.input : undefined }); + results.push({ + message: { + role: "tool", + content: [{ type: "tool_result", toolCallId: part.callID, toolName: part.tool, status: result.status, text: result.text }], + metadata: { ...meta, ocPartId: String(part.id ?? "") }, + }, + binding: { kind: "tool", messageIndex, partIndex }, + }); + } + }); + out.push({ + role: "assistant", + content: blocks, + metadata: { ...meta, ...(reasoning.length > 0 ? { reasoning } : {}), ...(message.info.summary ? { ocSummary: true } : {}) }, + }); + bindings.push({ kind: "assistant", messageIndex, textPartIndexes }); + for (const result of results) { + out.push(result.message); + bindings.push(result.binding); + } + }); + return { messages: out, bindings }; +} + +function textOf(message: RuntimeMessage | undefined): string[] { + if (!message || typeof message.content === "string") return message ? [String(message.content)] : []; + return message.content.flatMap((block) => (block.type === "text" || block.type === "tool_result" ? [block.text] : [])); +} + +/** + * Apply canonical changes back into `messages` IN PLACE at the array level + * (element replacement with clones), which is how OpenCode observes a + * `experimental.chat.messages.transform` result. Returns the number of + * replaced messages. + */ +export function applyOpenCodeChanges( + messages: OcMessageWithParts[], + decoded: OcDecoded, + canonical: readonly RuntimeMessage[], +): number { + const replacements = new Map(); + const current = (index: number): OcMessageWithParts => replacements.get(index) ?? messages[index]!; + + decoded.bindings.forEach((binding, canonicalIndex) => { + const before = textOf(decoded.messages[canonicalIndex]); + const after = textOf(canonical[canonicalIndex]); + if (after.length === before.length && after.every((text, i) => text === before[i])) return; + const message = current(binding.messageIndex); + const parts = message.parts.slice(); + + if (binding.kind === "tool") { + const part = parts[binding.partIndex] as OcToolPart; + const nextText = after[0] ?? ""; + if (part.state.status === "completed") { + if (part.state.time?.compacted) return; + parts[binding.partIndex] = { ...part, state: { ...part.state, output: nextText } }; + } else if (part.state.status === "error") { + const metadata = isRecord(part.state.metadata) ? part.state.metadata : undefined; + parts[binding.partIndex] = metadata?.interrupted === true && typeof metadata.output === "string" + ? { ...part, state: { ...part.state, metadata: { ...metadata, output: nextText } } } + : { ...part, state: { ...part.state, error: nextText } }; + } else { + return; + } + } else { + if (after.length === 0 || binding.textPartIndexes.length === 0) return; + // First text part carries the whole new text; the others are emptied (user + // text parts with empty text are skipped by OpenCode, assistant ones are ""). + binding.textPartIndexes.forEach((partIndex, i) => { + const part = parts[partIndex] as OcTextPart; + const text = i === 0 ? after.join("\n") : ""; + if (part.text !== text) parts[partIndex] = { ...part, text }; + }); + } + replacements.set(binding.messageIndex, { ...message, parts }); + }); + + for (const [index, next] of replacements) messages[index] = next; + return replacements.size; +} + +/* ------------------------------------------------------------------ * + * Eviction surface + * ------------------------------------------------------------------ */ + +/** Stable id for an eviction target: `part:` for tool results, `msg:` otherwise. */ +export function openCodeSurfaceId(message: RuntimeMessage): string { + const meta = message.metadata ?? {}; + if (message.role === "tool" && typeof meta.ocPartId === "string" && meta.ocPartId) return `part:${meta.ocPartId}`; + return `msg:${String(meta.ocMessageId ?? "")}`; +} + +/** Canonical messages → surface entries; turn N starts at the Nth user message. */ +export function openCodeSurfaceEntries( + canonical: readonly RuntimeMessage[], + evictedIds: ReadonlySet, +): CanonicalSurfaceEntry[] { + let turn = 0; + return canonical.map((message) => { + if (message.role === "user") turn += 1; + const id = openCodeSurfaceId(message); + const pruned = message.role === "tool" && textOf(message)[0] === OC_PRUNED_PLACEHOLDER; + return { + id, + turn, + message, + checkpoint: message.metadata?.ocSummary === true, + alreadyEvicted: evictedIds.has(id) || pruned, + }; + }); +} diff --git a/components/adapters/opencode/src/config.ts b/components/adapters/opencode/src/config.ts new file mode 100644 index 00000000..c8a31b88 --- /dev/null +++ b/components/adapters/opencode/src/config.ts @@ -0,0 +1,48 @@ +import { dirname, join, resolve } from "node:path"; +import { + loadCanonicalAdapterConfig, + normalizeCanonicalAdapterConfig, + runtimeHomeDir, + writeCanonicalAdapterConfig, + type CanonicalAdapterConfig, +} from "../../shared/canonical/config.js"; + +export type TokenPilotOpenCodeConfig = CanonicalAdapterConfig; + +/** + * OpenCode's global config dir: `$XDG_CONFIG_HOME/opencode`, default `~/.config/opencode` + * (core `global.ts`). `TOKENPILOT_OPENCODE_CONFIG_DIR` overrides it for tests/custom installs. + */ +export function defaultOpenCodeConfigDir(): string { + const override = process.env.TOKENPILOT_OPENCODE_CONFIG_DIR?.trim(); + if (override) return resolve(override); + const xdg = process.env.XDG_CONFIG_HOME?.trim(); + return join(xdg ? resolve(xdg) : join(runtimeHomeDir(), ".config"), "opencode"); +} + +export function defaultTokenPilotOpenCodeConfigPath(): string { + return process.env.TOKENPILOT_OPENCODE_CONFIG + ? resolve(process.env.TOKENPILOT_OPENCODE_CONFIG) + : join(defaultOpenCodeConfigDir(), "tokenpilot.json"); +} + +export function defaultOpenCodeStateDir(configPath = defaultTokenPilotOpenCodeConfigPath()): string { + return join(dirname(configPath), "tokenpilot-state", "tokenpilot"); +} + +export function normalizeTokenPilotOpenCodeConfig(raw: unknown, options?: { configPath?: string }): TokenPilotOpenCodeConfig { + return normalizeCanonicalAdapterConfig(raw, { + defaultStateDir: defaultOpenCodeStateDir(options?.configPath ?? defaultTokenPilotOpenCodeConfigPath()), + }); +} + +export function loadTokenPilotOpenCodeConfig(configPath = defaultTokenPilotOpenCodeConfigPath()): Promise { + return loadCanonicalAdapterConfig(configPath, { defaultStateDir: defaultOpenCodeStateDir(configPath) }); +} + +export function writeTokenPilotOpenCodeConfig( + config: TokenPilotOpenCodeConfig, + configPath = defaultTokenPilotOpenCodeConfigPath(), +): Promise { + return writeCanonicalAdapterConfig(config, configPath); +} diff --git a/components/adapters/opencode/src/doctor.ts b/components/adapters/opencode/src/doctor.ts new file mode 100644 index 00000000..ee237008 --- /dev/null +++ b/components/adapters/opencode/src/doctor.ts @@ -0,0 +1,222 @@ +/** + * `lightrsi opencode doctor` / `npm run doctor:opencode`. + */ +import { spawnSync } from "node:child_process"; +import { accessSync, constants, existsSync, readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { readLatestUxEffect, resolveLatestSessionId } from "@lightrsi/host-adapter"; +import { probeTokenPilotMcpServer, type TokenPilotMcpProbeResult } from "@lightrsi/mcp"; +import { canonicalRecoveryProtocolText } from "../../shared/canonical/before-call.js"; +import { evaluateEvictionReadiness, type EvictionReadiness } from "../../shared/canonical/config.js"; +import { adapterLogPath } from "../../shared/canonical/logger.js"; +import type { TokenPilotOpenCodeConfig } from "./config.js"; +import { + OPENCODE_MCP_KEY, + buildOpenCodeMcpEntry, + isTokenPilotOpenCodeLoader, + resolveOpenCodeInstallPaths, + type OpenCodeMcpEntry, +} from "./install.js"; +import { OPENCODE_TOKENPILOT_HOST_BINDING, openCodeSupportsEviction } from "./preset.js"; + +export type OpenCodeDoctorReport = { + configPath: string; + configExists: boolean; + enabled: boolean; + opencodeVersion?: string; + loaderPath: string; + loaderInstalled: boolean; + loaderBundlePath?: string; + bundleExists: boolean; + opencodeConfigPath: string; + opencodeConfigState: "json" | "jsonc_only" | "missing" | "unparseable"; + mcpRegistered: boolean; + mcpMatchesExpected: boolean; + mcpServerBuilt: boolean; + mcpProbe?: TokenPilotMcpProbeResult; + nativePrune: "on" | "off"; + stateDir: string; + stateDirWritable: boolean; + declaredFeatures: readonly string[]; + stabilizer: boolean; + dynamicContextTarget: "developer" | "user"; + reduction: boolean; + recoveryProtocolInjected: boolean; + eviction: EvictionReadiness; + latestSessionId?: string; + latestActivityAt?: string; + adapterLogExists: boolean; + healthy: boolean; + warnings: string[]; + problems: string[]; +}; + +/** The bundle path the installed ESM loader re-exports. */ +export function openCodeLoaderTarget(loaderPath: string): string | undefined { + try { + const literal = /from ("(?:[^"\\]|\\.)*");/.exec(readFileSync(loaderPath, "utf8"))?.[1]; + return literal ? fileURLToPath(JSON.parse(literal) as string) : undefined; + } catch { + return undefined; + } +} + +function writable(path: string): boolean { + try { + accessSync(path, constants.W_OK); + return true; + } catch { + return false; + } +} + +function detectOpenCodeVersion(): string | undefined { + try { + const out = spawnSync("opencode", ["--version"], { encoding: "utf8", timeout: 10_000 }); + const text = `${out.stdout ?? ""}`.trim(); + return out.status === 0 && text ? text.split("\n")[0] : undefined; + } catch { + return undefined; + } +} + +function sameEntry(observed: unknown, expected: OpenCodeMcpEntry): boolean { + if (!observed || typeof observed !== "object") return false; + const o = observed as Record; + return o.type === "local" + && Array.isArray(o.command) + && JSON.stringify(o.command) === JSON.stringify(expected.command) + && (o.environment as Record | undefined)?.TOKENPILOT_STATE_DIR === expected.environment.TOKENPILOT_STATE_DIR + && o.enabled !== false; +} + +export async function inspectOpenCodeDoctor(params: { + config: TokenPilotOpenCodeConfig; + configPath: string; + configDir?: string; + detectVersion?: boolean; + probeMcp?: boolean; +}): Promise { + const paths = resolveOpenCodeInstallPaths({ configDir: params.configDir, tokenPilotConfigPath: params.configPath }); + const loaderInstalled = await isTokenPilotOpenCodeLoader(paths.loaderPath); + const loaderBundlePath = loaderInstalled ? openCodeLoaderTarget(paths.loaderPath) : undefined; + const bundleExists = Boolean(loaderBundlePath && existsSync(loaderBundlePath)); + + let opencodeConfigState: OpenCodeDoctorReport["opencodeConfigState"] = "missing"; + let root: Record | undefined; + if (existsSync(paths.opencodeJsonPath)) { + try { + const parsed = JSON.parse(readFileSync(paths.opencodeJsonPath, "utf8")); + root = parsed && typeof parsed === "object" ? parsed as Record : undefined; + opencodeConfigState = root ? "json" : "unparseable"; + } catch { + opencodeConfigState = "unparseable"; + } + } else if (existsSync(paths.opencodeJsoncPath)) { + opencodeConfigState = "jsonc_only"; + } + const expected = buildOpenCodeMcpEntry(params.config.stateDir); + const observedEntry = (root?.mcp as Record | undefined)?.[OPENCODE_MCP_KEY]; + const mcpRegistered = observedEntry !== undefined; + const mcpMatchesExpected = sameEntry(observedEntry, expected.entry); + const compaction = root?.compaction as Record | undefined; + const nativePrune = compaction?.prune === true && !process.env.OPENCODE_DISABLE_PRUNE ? "on" : "off"; + const mcpProbe = params.probeMcp === false || !expected.built + ? undefined + : await probeTokenPilotMcpServer({ + serverName: OPENCODE_MCP_KEY, + command: expected.entry.command[0]!, + args: expected.entry.command.slice(1), + env: expected.entry.environment, + entryPath: expected.entryPath, + }, { timeoutMs: 15_000 }); + + const stateDir = params.config.stateDir; + const stateDirWritable = existsSync(stateDir) && writable(stateDir); + const eviction = evaluateEvictionReadiness(params.config, openCodeSupportsEviction()); + const latestSessionId = await resolveLatestSessionId(stateDir).catch(() => undefined); + const latest = await readLatestUxEffect(stateDir).catch(() => null); + + const problems: string[] = []; + const warnings: string[] = []; + const configExists = existsSync(params.configPath); + if (!configExists) problems.push(`config missing: ${params.configPath} (run install:opencode)`); + if (!params.config.enabled) problems.push("adapter disabled in config (enabled=false)"); + if (!loaderInstalled) problems.push(`plugin loader missing: ${paths.loaderPath} (run install:opencode)`); + if (loaderInstalled && !bundleExists) problems.push(`loader points to a missing bundle: ${loaderBundlePath ?? "(unparseable)"} (run build)`); + if (!mcpRegistered) { + problems.push(opencodeConfigState === "jsonc_only" + ? `recovery MCP not registered: only ${paths.opencodeJsoncPath} exists; paste the snippet printed by install:opencode` + : `recovery MCP not registered in ${paths.opencodeJsonPath}`); + } else if (!mcpMatchesExpected) { + problems.push("recovery MCP entry does not match the expected command/state dir (re-run install:opencode)"); + } + if (!expected.built) problems.push(`recovery MCP server not built: ${expected.entryPath} (pnpm --filter @lightrsi/mcp build)`); + if (mcpProbe && !mcpProbe.ok) problems.push(`recovery MCP probe failed: ${mcpProbe.detail}`); + if (!stateDirWritable) problems.push(`state dir not writable: ${stateDir}`); + if (params.config.modules.eviction && !eviction.active) problems.push(`eviction requested but inactive: ${eviction.reason}${eviction.missing.length ? ` (missing ${eviction.missing.join(", ")})` : ""}`); + if (params.config.hooks.dynamicContextTarget === "user") warnings.push("dynamicContextTarget=user is not supported on OpenCode; developer is used"); + if (nativePrune === "on") warnings.push("OpenCode native pruning (compaction.prune) is on: older tool outputs are cleared persistently by OpenCode"); + + return { + configPath: params.configPath, + configExists, + enabled: params.config.enabled, + ...(params.detectVersion === false ? {} : { opencodeVersion: detectOpenCodeVersion() }), + loaderPath: paths.loaderPath, + loaderInstalled, + ...(loaderBundlePath ? { loaderBundlePath } : {}), + bundleExists, + opencodeConfigPath: paths.opencodeJsonPath, + opencodeConfigState, + mcpRegistered, + mcpMatchesExpected, + mcpServerBuilt: expected.built, + ...(mcpProbe ? { mcpProbe } : {}), + nativePrune, + stateDir, + stateDirWritable, + declaredFeatures: OPENCODE_TOKENPILOT_HOST_BINDING.supportedFeatures, + stabilizer: params.config.modules.stabilizer, + dynamicContextTarget: params.config.hooks.dynamicContextTarget, + reduction: params.config.modules.reduction, + recoveryProtocolInjected: canonicalRecoveryProtocolText().length > 0, + eviction, + ...(latestSessionId ? { latestSessionId } : {}), + ...(typeof latest?.at === "string" ? { latestActivityAt: latest.at } : {}), + adapterLogExists: existsSync(adapterLogPath(stateDir)), + healthy: problems.length === 0, + warnings, + problems, + }; +} + +const yesNo = (value: boolean) => (value ? "yes" : "no"); + +export function formatOpenCodeDoctorReport(report: OpenCodeDoctorReport): string { + return [ + "TokenPilot OpenCode doctor:", + `- tokenpilot config: ${report.configPath} (${report.configExists ? "present" : "missing"})`, + `- enabled: ${yesNo(report.enabled)}`, + `- opencode version: ${report.opencodeVersion ?? "(opencode not found on PATH)"}`, + `- plugin loader: ${report.loaderPath} (${report.loaderInstalled ? "installed" : "missing"})`, + `- plugin bundle: ${report.loaderBundlePath ?? "(unknown)"} (${report.bundleExists ? "present" : "missing"})`, + `- opencode config: ${report.opencodeConfigPath} (${report.opencodeConfigState})`, + `- recovery MCP registered: ${yesNo(report.mcpRegistered)} (matches expected: ${yesNo(report.mcpMatchesExpected)})`, + `- recovery MCP server built: ${yesNo(report.mcpServerBuilt)}`, + `- recovery MCP probe: ${report.mcpProbe ? (report.mcpProbe.ok ? "ok" : `failed (${report.mcpProbe.detail})`) : "(skipped)"}`, + `- native tool-output pruning: ${report.nativePrune}`, + `- stateDir: ${report.stateDir} (${report.stateDirWritable ? "writable" : "not writable"})`, + `- declared features: ${report.declaredFeatures.join(", ")}`, + `- stabilizer: ${yesNo(report.stabilizer)} (target ${report.dynamicContextTarget === "user" ? "user → developer (unsupported)" : "developer"})`, + `- reduction: ${yesNo(report.reduction)}`, + `- recovery protocol injected: ${yesNo(report.recoveryProtocolInjected)}`, + `- eviction: ${report.eviction.active ? "active" : `inactive (${report.eviction.reason})`}`, + `- latest session: ${report.latestSessionId ?? "(none yet — start an OpenCode session)"}`, + `- latest activity: ${report.latestActivityAt ?? "(none yet)"}`, + `- adapter log: ${report.adapterLogExists ? adapterLogPath(report.stateDir) : "(none)"}`, + `- healthy: ${yesNo(report.healthy)}`, + ...report.warnings.map((warning) => `- warning: ${warning}`), + ...report.problems.map((problem) => `- problem: ${problem}`), + ].join("\n"); +} diff --git a/components/adapters/opencode/src/host-config-adapter.ts b/components/adapters/opencode/src/host-config-adapter.ts new file mode 100644 index 00000000..62cbffe2 --- /dev/null +++ b/components/adapters/opencode/src/host-config-adapter.ts @@ -0,0 +1,15 @@ +import type { ProductSurfaceConfigAdapter } from "@lightrsi/host-adapter"; + +export function resolveOpenCodeStateDir(config: Record): string | undefined { + const stateDir = config.stateDir; + return typeof stateDir === "string" && stateDir.trim().length > 0 ? stateDir.trim() : undefined; +} + +/** OpenCode's runtime config is flat, like Claude Code's `tokenpilot.json`. */ +export const openCodeProductSurfaceConfigAdapter: ProductSurfaceConfigAdapter = { + pluginConfigRecord: (config) => config, + pluginEntryRecord: (config) => config, + ensurePluginConfig: (config) => config, + ensurePluginEntry: (config) => config, + resolveStateDir: resolveOpenCodeStateDir, +}; diff --git a/components/adapters/opencode/src/install.ts b/components/adapters/opencode/src/install.ts new file mode 100644 index 00000000..f3b7197d --- /dev/null +++ b/components/adapters/opencode/src/install.ts @@ -0,0 +1,323 @@ +/** + * Reversible install for the OpenCode adapter. + * + * install: + * 1. writes a marker-tagged ESM loader `/plugins/tokenpilot.js` that + * re-exports the built `dist/plugin.mjs` (OpenCode auto-loads `plugins/*.js`); + * a foreign file at that path is moved to a timestamped backup first; + * 2. registers the shared recovery MCP server as `mcp.tokenpilot_memory_fault_recover` + * in `/opencode.json` after writing a timestamped backup of the file. + * An `opencode.jsonc`-only setup (or unparseable JSON) is never rewritten: the + * snippet is returned for the user to paste; + * 3. writes `/tokenpilot.json` in normal mode if missing (never credentials). + * uninstall removes exactly what install added (loader, the one `mcp` key) and restores + * the loader backup; `purge` also removes the created config and state. + */ +import { existsSync } from "node:fs"; +import { copyFile, mkdir, readFile, rename, rm, rmdir, writeFile } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; +import { resolveTokenPilotMcpServerSpec, TOKENPILOT_MCP_SERVER_NAME } from "@lightrsi/mcp"; +import { installLightRsiCliBin } from "../../shared/cli-bin-install.js"; +import { rememberCliHostPathOverrides } from "../../shared/cli-context.js"; +import { + defaultOpenCodeConfigDir, + loadTokenPilotOpenCodeConfig, + normalizeTokenPilotOpenCodeConfig, + writeTokenPilotOpenCodeConfig, +} from "./config.js"; + +export const OPENCODE_LOADER_MARKER = "// tokenpilot-managed-loader: opencode"; +export const OPENCODE_MCP_KEY = TOKENPILOT_MCP_SERVER_NAME; + +export type OpenCodeMcpEntry = { + type: "local"; + command: string[]; + environment: Record; + enabled: boolean; +}; + +export type OpenCodeInstallManifest = { + version: 1; + loaderPath: string; + bundlePath: string; + loaderBackupPath?: string; + opencodeConfigPath: string; + opencodeConfigCreated: boolean; + opencodeConfigBackupPath?: string; + mcpRegistered: boolean; + tokenPilotConfigPath: string; + tokenPilotConfigCreated: boolean; + installedAt: string; +}; + +export type OpenCodeInstallPaths = { + configDir: string; + loaderPath: string; + opencodeJsonPath: string; + opencodeJsoncPath: string; + tokenPilotConfigPath: string; + manifestPath: string; +}; + +export function resolveOpenCodeInstallPaths(params?: { configDir?: string; tokenPilotConfigPath?: string }): OpenCodeInstallPaths { + const configDir = params?.configDir ? resolve(params.configDir) : defaultOpenCodeConfigDir(); + return { + configDir, + loaderPath: join(configDir, "plugins", "tokenpilot.js"), + opencodeJsonPath: join(configDir, "opencode.json"), + opencodeJsoncPath: join(configDir, "opencode.jsonc"), + tokenPilotConfigPath: params?.tokenPilotConfigPath ? resolve(params.tokenPilotConfigPath) : join(configDir, "tokenpilot.json"), + manifestPath: join(configDir, "tokenpilot-install.json"), + }; +} + +export function openCodeAdapterRoot(moduleDir = __dirname): string { + let current = resolve(moduleDir); + for (let i = 0; i < 6; i += 1) { + if (existsSync(join(current, "package.json")) && existsSync(join(current, "src", "plugin.ts"))) return current; + current = dirname(current); + } + return resolve(moduleDir, ".."); +} + +export function renderOpenCodeLoader(bundlePath: string): string { + return [ + OPENCODE_LOADER_MARKER, + "// Installed by `npm --prefix components/adapters/opencode run install:opencode`. Remove with `uninstall:opencode`.", + `export { default } from ${JSON.stringify(pathToFileURL(bundlePath).href)};`, + "", + ].join("\n"); +} + +export async function isTokenPilotOpenCodeLoader(path: string): Promise { + if (!existsSync(path)) return false; + try { + return (await readFile(path, "utf8")).startsWith(OPENCODE_LOADER_MARKER); + } catch { + return false; + } +} + +export function buildOpenCodeMcpEntry(stateDir: string): { entry: OpenCodeMcpEntry; entryPath: string; built: boolean } { + const spec = resolveTokenPilotMcpServerSpec({ stateDir, requireBuild: false }); + return { + entry: { type: "local", command: [spec.command, ...spec.args], environment: { ...spec.env }, enabled: true }, + entryPath: spec.entryPath, + built: existsSync(spec.entryPath), + }; +} + +async function readManifest(path: string): Promise { + try { + const parsed = JSON.parse(await readFile(path, "utf8")) as OpenCodeInstallManifest; + return parsed?.version === 1 ? parsed : undefined; + } catch { + return undefined; + } +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +export async function installOpenCodeTokenPilot(params?: { + configDir?: string; + tokenPilotConfigPath?: string; + bundlePath?: string; + cliBinDir?: string; + cliContextPath?: string; + installCliBin?: boolean; + now?: () => Date; +}): Promise<{ + paths: OpenCodeInstallPaths; + bundlePath: string; + loaderBackupPath?: string; + tokenPilotConfigCreated: boolean; + stateDir: string; + mcp: { + registered: boolean; + reason: "registered" | "updated" | "jsonc_only" | "unparseable"; + entry: OpenCodeMcpEntry; + serverBuilt: boolean; + opencodeConfigCreated: boolean; + backupPath?: string; + snippet: string; + }; + cliBin?: Awaited>; +}> { + const paths = resolveOpenCodeInstallPaths(params); + const now = params?.now ?? (() => new Date()); + const bundlePath = params?.bundlePath ? resolve(params.bundlePath) : join(openCodeAdapterRoot(), "dist", "plugin.mjs"); + if (!existsSync(bundlePath)) { + throw new Error(`OpenCode adapter bundle not built: ${bundlePath}. Run \`npm --prefix components/adapters/opencode run build\` first.`); + } + const previous = await readManifest(paths.manifestPath); + + // 1. Plugin loader. + let loaderBackupPath = previous?.loaderBackupPath; + if (existsSync(paths.loaderPath) && !(await isTokenPilotOpenCodeLoader(paths.loaderPath))) { + loaderBackupPath = `${paths.loaderPath}.bak-${now().getTime()}`; + await rename(paths.loaderPath, loaderBackupPath); + } + await mkdir(dirname(paths.loaderPath), { recursive: true }); + await writeFile(paths.loaderPath, renderOpenCodeLoader(bundlePath), "utf8"); + + // 3 (first, so the MCP entry can carry the final state dir). TokenPilot config. + const tokenPilotConfigCreated = !existsSync(paths.tokenPilotConfigPath); + const config = tokenPilotConfigCreated + ? normalizeTokenPilotOpenCodeConfig({}, { configPath: paths.tokenPilotConfigPath }) + : await loadTokenPilotOpenCodeConfig(paths.tokenPilotConfigPath); + if (tokenPilotConfigCreated) await writeTokenPilotOpenCodeConfig(config, paths.tokenPilotConfigPath); + await mkdir(config.stateDir, { recursive: true }); + + // 2. Recovery MCP registration. + const mcpSpec = buildOpenCodeMcpEntry(config.stateDir); + const snippet = JSON.stringify({ mcp: { [OPENCODE_MCP_KEY]: mcpSpec.entry } }, null, 2); + let mcp: Awaited>["mcp"]; + const jsonExists = existsSync(paths.opencodeJsonPath); + if (!jsonExists && existsSync(paths.opencodeJsoncPath)) { + mcp = { registered: false, reason: "jsonc_only", entry: mcpSpec.entry, serverBuilt: mcpSpec.built, opencodeConfigCreated: false, snippet }; + } else { + let root: Record | undefined = { $schema: "https://opencode.ai/config.json" }; + if (jsonExists) { + try { + const parsed = JSON.parse(await readFile(paths.opencodeJsonPath, "utf8")); + root = isRecord(parsed) ? parsed : undefined; + } catch { + root = undefined; + } + } + if (!root) { + mcp = { registered: false, reason: "unparseable", entry: mcpSpec.entry, serverBuilt: mcpSpec.built, opencodeConfigCreated: false, snippet }; + } else { + let backupPath: string | undefined; + if (jsonExists) { + backupPath = `${paths.opencodeJsonPath}.tokenpilot-backup-${now().getTime()}`; + await copyFile(paths.opencodeJsonPath, backupPath); + } + const existingMcp = isRecord(root.mcp) ? root.mcp : {}; + const updated = OPENCODE_MCP_KEY in existingMcp; + root = { ...root, mcp: { ...existingMcp, [OPENCODE_MCP_KEY]: mcpSpec.entry } }; + await mkdir(paths.configDir, { recursive: true }); + await writeFile(paths.opencodeJsonPath, `${JSON.stringify(root, null, 2)}\n`, "utf8"); + mcp = { + registered: true, + reason: updated ? "updated" : "registered", + entry: mcpSpec.entry, + serverBuilt: mcpSpec.built, + opencodeConfigCreated: !jsonExists, + ...(backupPath ? { backupPath } : {}), + snippet, + }; + } + } + + const manifest: OpenCodeInstallManifest = { + version: 1, + loaderPath: paths.loaderPath, + bundlePath, + ...(loaderBackupPath ? { loaderBackupPath } : {}), + opencodeConfigPath: paths.opencodeJsonPath, + opencodeConfigCreated: mcp.opencodeConfigCreated || previous?.opencodeConfigCreated === true, + ...(mcp.backupPath ? { opencodeConfigBackupPath: mcp.backupPath } : previous?.opencodeConfigBackupPath ? { opencodeConfigBackupPath: previous.opencodeConfigBackupPath } : {}), + mcpRegistered: mcp.registered || previous?.mcpRegistered === true, + tokenPilotConfigPath: paths.tokenPilotConfigPath, + tokenPilotConfigCreated: tokenPilotConfigCreated || previous?.tokenPilotConfigCreated === true, + installedAt: now().toISOString(), + }; + await writeFile(paths.manifestPath, `${JSON.stringify(manifest, null, 2)}\n`, "utf8"); + + await rememberCliHostPathOverrides("opencode", { + tokenPilotConfigPath: paths.tokenPilotConfigPath, + hostConfigPath: paths.opencodeJsonPath, + }, params?.cliContextPath); + const cliBin = params?.installCliBin === false + ? undefined + : await installLightRsiCliBin({ adapterRoot: openCodeAdapterRoot(), binDir: params?.cliBinDir }); + + return { + paths, + bundlePath, + ...(loaderBackupPath ? { loaderBackupPath } : {}), + tokenPilotConfigCreated, + stateDir: config.stateDir, + mcp, + ...(cliBin ? { cliBin } : {}), + }; +} + +export async function uninstallOpenCodeTokenPilot(params?: { + configDir?: string; + tokenPilotConfigPath?: string; + purge?: boolean; +}): Promise<{ + loaderRemoved: boolean; + loaderBackupRestored: boolean; + foreignLoaderKept: boolean; + mcpRemoved: boolean; + opencodeConfigRemoved: boolean; + configRemoved: boolean; + stateRemoved: boolean; +}> { + const paths = resolveOpenCodeInstallPaths(params); + const manifest = await readManifest(paths.manifestPath); + + let loaderRemoved = false; + let foreignLoaderKept = false; + if (await isTokenPilotOpenCodeLoader(paths.loaderPath)) { + await rm(paths.loaderPath); + loaderRemoved = true; + } else if (existsSync(paths.loaderPath)) { + foreignLoaderKept = true; + } + let loaderBackupRestored = false; + if (manifest?.loaderBackupPath && existsSync(manifest.loaderBackupPath) && !existsSync(paths.loaderPath)) { + await rename(manifest.loaderBackupPath, paths.loaderPath); + loaderBackupRestored = true; + } + + let mcpRemoved = false; + let opencodeConfigRemoved = false; + if (manifest?.mcpRegistered && existsSync(paths.opencodeJsonPath)) { + try { + const root = JSON.parse(await readFile(paths.opencodeJsonPath, "utf8")) as Record; + if (isRecord(root.mcp) && OPENCODE_MCP_KEY in root.mcp) { + const { [OPENCODE_MCP_KEY]: _removed, ...restMcp } = root.mcp; + const next: Record = { ...root }; + if (Object.keys(restMcp).length > 0) next.mcp = restMcp; + else delete next.mcp; + mcpRemoved = true; + const onlySchemaLeft = Object.keys(next).every((key) => key === "$schema"); + if (manifest.opencodeConfigCreated && onlySchemaLeft) { + await rm(paths.opencodeJsonPath); + opencodeConfigRemoved = true; + } else { + await writeFile(paths.opencodeJsonPath, `${JSON.stringify(next, null, 2)}\n`, "utf8"); + } + } + } catch { + // Leave an unparseable config alone; doctor will report the stale entry. + } + } + + let configRemoved = false; + let stateRemoved = false; + if (params?.purge) { + const configPath = manifest?.tokenPilotConfigPath ?? paths.tokenPilotConfigPath; + const stateDir = existsSync(configPath) ? (await loadTokenPilotOpenCodeConfig(configPath)).stateDir : undefined; + if (manifest?.tokenPilotConfigCreated && existsSync(configPath)) { + await rm(configPath); + configRemoved = true; + } + if (stateDir && existsSync(stateDir)) { + await rm(stateDir, { recursive: true, force: true }); + stateRemoved = true; + const parent = dirname(stateDir); + // rmdir only succeeds on an empty directory, so a shared parent is never removed. + if (parent.endsWith("tokenpilot-state")) await rmdir(parent).catch(() => undefined); + } + } + await rm(paths.manifestPath, { force: true }); + return { loaderRemoved, loaderBackupRestored, foreignLoaderKept, mcpRemoved, opencodeConfigRemoved, configRemoved, stateRemoved }; +} diff --git a/components/adapters/opencode/src/opencode-types.ts b/components/adapters/opencode/src/opencode-types.ts new file mode 100644 index 00000000..9b635391 --- /dev/null +++ b/components/adapters/opencode/src/opencode-types.ts @@ -0,0 +1,79 @@ +/** + * Minimal structural mirrors of the OpenCode v1 plugin API this adapter touches. + * + * Like the DeepSeek Harness adapter, the adapter does not import the + * host's packages: OpenCode loads it in-process (Bun) and passes objects that + * match these shapes. Mirrored from the pinned `@opencode-ai/plugin@1.18.33` + * (`dist/index.d.ts`, `Hooks`) and `@opencode-ai/sdk@1.18.33` + * (`dist/gen/types.gen.d.ts`, `Message` / `Part`). + */ + +export type OcTextPart = { + id: string; sessionID: string; messageID: string; type: "text"; text: string; + synthetic?: boolean; ignored?: boolean; [key: string]: unknown; +}; +export type OcReasoningPart = { id: string; sessionID: string; messageID: string; type: "reasoning"; text: string; [key: string]: unknown }; +export type OcFilePart = { id: string; sessionID: string; messageID: string; type: "file"; mime: string; url: string; filename?: string }; +export type OcToolState = + | { status: "pending" | "running"; input: Record; [key: string]: unknown } + | { + status: "completed"; + input: Record; + output: string; + title: string; + metadata: Record; + time: { start: number; end: number; compacted?: number }; + attachments?: OcFilePart[]; + } + | { + status: "error"; + input: Record; + error: string; + metadata?: Record; + time: { start: number; end: number }; + }; +export type OcToolPart = { + id: string; sessionID: string; messageID: string; type: "tool"; callID: string; tool: string; state: OcToolState; + [key: string]: unknown; +}; +export type OcOtherPart = { id: string; sessionID?: string; messageID?: string; type: string; [key: string]: unknown }; +export type OcPart = OcTextPart | OcReasoningPart | OcFilePart | OcToolPart | OcOtherPart; + +export type OcMessageInfo = { + id: string; + sessionID: string; + role: "user" | "assistant"; + /** Assistant compaction summaries set this. */ + summary?: unknown; + modelID?: string; + providerID?: string; + error?: unknown; + [key: string]: unknown; +}; + +export type OcMessageWithParts = { info: OcMessageInfo; parts: OcPart[] }; + +export type OcModel = { id?: string; providerID?: string; [key: string]: unknown }; + +export type OcPluginInput = { + directory: string; + worktree: string; + [key: string]: unknown; +}; + +export type OcHooks = { + "experimental.chat.system.transform"?: ( + input: { sessionID?: string; model: OcModel }, + output: { system: string[] }, + ) => Promise; + "experimental.chat.messages.transform"?: ( + input: Record, + output: { messages: OcMessageWithParts[] }, + ) => Promise; + dispose?: () => Promise; +}; + +export type OcPluginModule = { + id: string; + server: (input: OcPluginInput, options?: Record) => Promise; +}; diff --git a/components/adapters/opencode/src/overlay.ts b/components/adapters/opencode/src/overlay.ts new file mode 100644 index 00000000..8b5de44e --- /dev/null +++ b/components/adapters/opencode/src/overlay.ts @@ -0,0 +1,55 @@ +/** + * Durable eviction overlay for OpenCode. + * + * OpenCode 1.18.33 gives plugins no persistent part-edit hook, so lifecycle + * eviction is applied request-locally in `experimental.chat.messages.transform` + * (the Claude Code adapter's overlay approach). The set of evicted targets and + * their replacement text is stored per session so every later request re-applies + * exactly the same bytes: eviction decisions are durable and the outgoing + * history stays prefix-stable. + */ +import { readFile } from "node:fs/promises"; +import { join } from "node:path"; +import { writeJsonFileAtomic } from "@lightrsi/host-adapter"; +import type { RuntimeMessage } from "@lightrsi/kernel"; +import { openCodeSurfaceId } from "./codec.js"; + +export type OverlayEntry = { text: string; at: string }; +export type EvictionOverlay = { version: 1; sessionId: string; entries: Record }; + +export function overlayPath(stateDir: string, sessionId: string): string { + return join(stateDir, "tokenpilot", "eviction-overlay", `${encodeURIComponent(sessionId)}.json`); +} + +export async function loadOverlay(stateDir: string, sessionId: string): Promise { + try { + const parsed = JSON.parse(await readFile(overlayPath(stateDir, sessionId), "utf8")) as EvictionOverlay; + if (parsed && parsed.version === 1 && parsed.entries && typeof parsed.entries === "object") return parsed; + } catch { + // Missing or unreadable overlay: start empty (nothing is evicted). + } + return { version: 1, sessionId, entries: {} }; +} + +export async function saveOverlay(stateDir: string, overlay: EvictionOverlay): Promise { + await writeJsonFileAtomic(overlayPath(stateDir, overlay.sessionId), overlay); +} + +/** Replace overlaid messages' text with their stored replacement (tool_call blocks are kept). */ +export function applyOverlay(messages: readonly RuntimeMessage[], overlay: EvictionOverlay): { messages: RuntimeMessage[]; applied: number } { + let applied = 0; + const next = messages.map((message) => { + const entry = overlay.entries[openCodeSurfaceId(message)]; + if (!entry || !Array.isArray(message.content)) return message; + applied += 1; + if (message.role === "tool") { + return { + ...message, + content: message.content.map((block) => (block.type === "tool_result" ? { ...block, text: entry.text } : block)), + }; + } + const kept = message.content.filter((block) => block.type !== "text"); + return { ...message, content: [{ type: "text" as const, text: entry.text }, ...kept] }; + }); + return { messages: next, applied }; +} diff --git a/components/adapters/opencode/src/plugin.ts b/components/adapters/opencode/src/plugin.ts new file mode 100644 index 00000000..dd4598e4 --- /dev/null +++ b/components/adapters/opencode/src/plugin.ts @@ -0,0 +1,18 @@ +/** + * OpenCode v1 plugin entry (`~/.config/opencode/plugins/tokenpilot.js` loads this). + * + * OpenCode reads `default` as `{ id, server }` (`plugin/shared.ts` `readV1Plugin`). + * Only the default export exists: OpenCode's legacy loader would reject any other + * non-function export. + */ +import type { OcPluginModule } from "./opencode-types.js"; +import { OpenCodeTokenPilotRuntime } from "./runtime.js"; + +const plugin: OcPluginModule = { + id: "tokenpilot", + async server() { + return new OpenCodeTokenPilotRuntime().hooks(); + }, +}; + +export default plugin; diff --git a/components/adapters/opencode/src/preset.ts b/components/adapters/opencode/src/preset.ts new file mode 100644 index 00000000..3c515845 --- /dev/null +++ b/components/adapters/opencode/src/preset.ts @@ -0,0 +1,17 @@ +import { + createTokenPilotHostBinding, + initializeTokenPilotPreset, +} from "@lightrsi/tokenpilot"; + +export const OPENCODE_TOKENPILOT_HOST_BINDING = createTokenPilotHostBinding({ + hostId: "opencode", + supportedFeatures: ["stabilizer", "reduction", "eviction"], +}); + +export function initializeOpenCodeTokenPilotPreset(): void { + initializeTokenPilotPreset(OPENCODE_TOKENPILOT_HOST_BINDING); +} + +export function openCodeSupportsEviction(): boolean { + return OPENCODE_TOKENPILOT_HOST_BINDING.supportedFeatures.includes("eviction"); +} diff --git a/components/adapters/opencode/src/product-registration.ts b/components/adapters/opencode/src/product-registration.ts new file mode 100644 index 00000000..20386700 --- /dev/null +++ b/components/adapters/opencode/src/product-registration.ts @@ -0,0 +1,17 @@ +import { readLatestUxEffect } from "@lightrsi/host-adapter"; +import { defineProductHostRegistration } from "@lightrsi/product-surface"; +import { defaultTokenPilotOpenCodeConfigPath, loadTokenPilotOpenCodeConfig } from "./config.js"; +import { OPENCODE_TOKENPILOT_HOST_BINDING } from "./preset.js"; + +export const OPENCODE_PRODUCT_HOST_REGISTRATION = defineProductHostRegistration({ + hostId: "opencode", + displayName: "OpenCode", + preset: OPENCODE_TOKENPILOT_HOST_BINDING, + async resolveStateDir(context) { + const config = await loadTokenPilotOpenCodeConfig( + context?.productConfigPath?.trim() || defaultTokenPilotOpenCodeConfigPath(), + ); + return config.stateDir; + }, + readLatestActivity: readLatestUxEffect, +}); diff --git a/components/adapters/opencode/src/runtime.ts b/components/adapters/opencode/src/runtime.ts new file mode 100644 index 00000000..82e251bf --- /dev/null +++ b/components/adapters/opencode/src/runtime.ts @@ -0,0 +1,225 @@ +/** + * TokenPilot runtime for OpenCode (v1 plugin hooks, 1.18.33): + * + * experimental.chat.system.transform stable prefix (shared stabilizer) + shared + * recovery protocol, per LLM request + * experimental.chat.messages.transform per model step: durable eviction overlay → + * opt-in lifecycle eviction → request-time + * reduction (shared before-call path) + * recovery shared recovery MCP server (registered by + * install in opencode.json) + * + * OpenCode does not catch plugin exceptions (`Plugin.trigger`), so every hook body + * is wrapped by `failOpen`, and host objects are only touched at the very end with + * cloned replacements. + */ +import { stat } from "node:fs/promises"; +import { configureStatePathResolver } from "@lightrsi/artifact-store"; +import type { TaskStateEstimator } from "@lightrsi/eviction"; +import type { SessionTaskRegistry } from "@lightrsi/history"; +import { + appendRecentTurnBinding, + createStaticStatePathResolver, + writeLatestSessionRef, +} from "@lightrsi/host-adapter"; +import type { RuntimeMessage } from "@lightrsi/kernel"; +import { + applyCanonicalStablePrefix, + createCanonicalEnvelope, + recordCanonicalStability, + runCanonicalBeforeCallReduction, + withCanonicalRecoveryProtocol, +} from "../../shared/canonical/before-call.js"; +import { evaluateEvictionReadiness } from "../../shared/canonical/config.js"; +import { createCanonicalEstimator, runCanonicalSurfaceEviction } from "../../shared/canonical/eviction.js"; +import { createFileLogger, failOpen, type AdapterLogger } from "../../shared/canonical/logger.js"; +import { loadReductionMemo, saveReductionMemoIfDirty } from "../../shared/canonical/memo-store.js"; +import { ReductionMemo, passSavedChars } from "../../shared/canonical/reduction.js"; +import { applyOpenCodeChanges, decodeOpenCodeMessages, openCodeSurfaceEntries } from "./codec.js"; +import { + defaultTokenPilotOpenCodeConfigPath, + loadTokenPilotOpenCodeConfig, + type TokenPilotOpenCodeConfig, +} from "./config.js"; +import type { OcHooks, OcMessageWithParts, OcModel } from "./opencode-types.js"; +import { applyOverlay, loadOverlay, saveOverlay } from "./overlay.js"; +import { initializeOpenCodeTokenPilotPreset, openCodeSupportsEviction } from "./preset.js"; + +export const OPENCODE_HOST_ID = "opencode"; +export const OPENCODE_DISPLAY_NAME = "OpenCode"; + +export type OpenCodeRuntimeDependencies = { + configPath?: string; + loadConfig?: () => Promise; + supportsEviction?: boolean; + createEstimator?: (config: TokenPilotOpenCodeConfig) => TaskStateEstimator | undefined; + registryStore?: { + load(sessionId: string): Promise | SessionTaskRegistry; + persist(registry: SessionTaskRegistry, expectedVersion: number): Promise | void; + }; +}; + +export class OpenCodeTokenPilotRuntime { + config: TokenPilotOpenCodeConfig | undefined; + readonly logger: AdapterLogger; + private readonly memos = new Map(); + private configStamp: string | undefined; + private warnedUserTarget = false; + private readonly deps: OpenCodeRuntimeDependencies & { configPath: string }; + + constructor(deps: OpenCodeRuntimeDependencies = {}) { + this.deps = { ...deps, configPath: deps.configPath ?? defaultTokenPilotOpenCodeConfigPath() }; + this.logger = createFileLogger({ + hostId: OPENCODE_HOST_ID, + stateDir: () => this.config?.stateDir, + debug: () => this.config?.logLevel === "debug", + }); + } + + /** Session memo, loaded from stateDir on first use so OpenCode restarts keep the prefix stable. */ + async memoFor(stateDir: string, sessionId: string): Promise { + let memo = this.memos.get(sessionId); + if (!memo) { + memo = await loadReductionMemo(stateDir, sessionId); + this.memos.set(sessionId, memo); + } + return memo; + } + + /** Load config once, then again whenever tokenpilot.json changes (CLI mode switches apply live). */ + async ensureConfig(): Promise { + if (this.deps.loadConfig) { + if (!this.config) this.bind(await this.deps.loadConfig()); + return this.config?.enabled ? this.config : undefined; + } + const stamp = await stat(this.deps.configPath).then((s) => `${s.mtimeMs}:${s.size}`).catch(() => "missing"); + if (!this.config || stamp !== this.configStamp) { + this.bind(await loadTokenPilotOpenCodeConfig(this.deps.configPath)); + this.configStamp = stamp; + } + return this.config?.enabled ? this.config : undefined; + } + + private bind(config: TokenPilotOpenCodeConfig): void { + this.config = config; + initializeOpenCodeTokenPilotPreset(); + configureStatePathResolver(createStaticStatePathResolver({ + hostId: OPENCODE_HOST_ID, + displayName: OPENCODE_DISPLAY_NAME, + stateDir: config.stateDir, + namespaceDir: "tokenpilot", + })); + } + + onSystemTransform(input: { sessionID?: string; model?: OcModel }, output: { system: string[] }): Promise { + return failOpen(this.logger, "experimental.chat.system.transform", async () => { + const config = await this.ensureConfig(); + if (!config || !Array.isArray(output?.system) || typeof output.system[0] !== "string") return; + if (config.hooks.dynamicContextTarget === "user" && !this.warnedUserTarget) { + this.warnedUserTarget = true; + this.logger.warn("dynamicContextTarget=user is not supported on OpenCode 1.18.33 (messages.transform runs before system.transform); using developer"); + } + const effective = { ...config, hooks: { ...config.hooks, dynamicContextTarget: "developer" as const } }; + const before = output.system[0]; + const stable = applyCanonicalStablePrefix(before, effective); + const withRecovery = withCanonicalRecoveryProtocol(stable.instructions); + if (withRecovery.text !== before) output.system[0] = withRecovery.text; + if (stable.changed && input?.sessionID) { + await recordCanonicalStability({ + config: effective, + sessionId: input.sessionID, + model: String(input.model?.id ?? ""), + before, + result: stable, + }).catch((error) => this.logger.warn("stability snapshot failed", error)); + } + }, undefined); + } + + onMessagesTransform(output: { messages: OcMessageWithParts[] }): Promise { + return failOpen(this.logger, "experimental.chat.messages.transform", async () => { + const config = await this.ensureConfig(); + const messages = output?.messages; + if (!config || !Array.isArray(messages) || messages.length === 0) return; + const sessionId = messages.map((m) => m?.info?.sessionID).find((id): id is string => typeof id === "string" && id.length > 0); + if (!sessionId) return; + const lastUser = [...messages].reverse().find((m) => m?.info?.role === "user"); + const model = String((lastUser?.info?.model as { modelID?: string } | undefined)?.modelID ?? ""); + + const decoded = decodeOpenCodeMessages(messages); + let canonical: RuntimeMessage[] = decoded.messages; + + let overlay = await loadOverlay(config.stateDir, sessionId); + canonical = applyOverlay(canonical, overlay).messages; + + // Eviction is isolated: an estimator outage must not also disable reduction. + try { + const readiness = evaluateEvictionReadiness(config, this.deps.supportsEviction ?? openCodeSupportsEviction()); + if (readiness.active) { + const estimator = (this.deps.createEstimator ?? createCanonicalEstimator)(config); + if (estimator) { + const entries = openCodeSurfaceEntries(canonical, new Set(Object.keys(overlay.entries))); + const cycle = await runCanonicalSurfaceEviction({ + sessionId, + entries, + config, + estimator, + registryStore: this.deps.registryStore, + apply: async (replacements) => { + const at = new Date().toISOString(); + const next = { ...overlay, entries: { ...overlay.entries } }; + for (const replacement of replacements) next.entries[replacement.id] = { text: replacement.text, at }; + try { + await saveOverlay(config.stateDir, next); + } catch (error) { + this.logger.warn("eviction overlay save failed", error); + return "deferred"; + } + overlay = next; + return "committed"; + }, + }); + if (cycle.status === "applied") { + canonical = applyOverlay(canonical, overlay).messages; + this.logger.info("eviction applied", { sessionId, targets: cycle.result.replacements.map((r) => r.id) }); + } + } + } + } catch (error) { + this.logger.warn("eviction failed open", error); + } + + let savedChars = 0; + let passSaved: Record = {}; + if (config.modules.reduction) { + const reduced = await runCanonicalBeforeCallReduction({ + envelope: createCanonicalEnvelope({ hostId: OPENCODE_HOST_ID, displayName: OPENCODE_DISPLAY_NAME, sessionId, model, messages: canonical }), + config, + memo: await this.memoFor(config.stateDir, sessionId), + }); + canonical = reduced.messages; + await saveReductionMemoIfDirty(config.stateDir, sessionId, await this.memoFor(config.stateDir, sessionId)) + .catch((error) => this.logger.warn("reduction memo save failed", error)); + savedChars = reduced.summary?.savedChars ?? 0; + passSaved = passSavedChars(reduced.summary); + } + + const changed = applyOpenCodeChanges(messages, decoded, canonical); + await writeLatestSessionRef(config.stateDir, sessionId, new Date().toISOString()); + if (savedChars > 0) { + await appendRecentTurnBinding(config.stateDir, { sessionId, updatedAt: new Date().toISOString(), model, reductionSavedChars: savedChars, reductionPassSavedChars: passSaved, changedMessages: changed }) + .catch((error) => this.logger.warn("turn binding failed", error)); + } + }, undefined); + } + + hooks(): OcHooks { + return { + "experimental.chat.system.transform": (input, output) => this.onSystemTransform(input, output), + "experimental.chat.messages.transform": (_input, output) => this.onMessagesTransform(output), + dispose: async () => { + this.memos.clear(); + }, + }; + } +} diff --git a/components/adapters/opencode/tests/binding-registration.test.ts b/components/adapters/opencode/tests/binding-registration.test.ts new file mode 100644 index 00000000..e60bcaf0 --- /dev/null +++ b/components/adapters/opencode/tests/binding-registration.test.ts @@ -0,0 +1,62 @@ +/** + * Regression matrix: opencode/src/preset.ts + product-registration.ts + config paths + * + * B1 host binding: hostId "opencode", TokenPilot preset id/version, features exactly + * [stabilizer, reduction, eviction]; initializing the preset does not throw + * B2 product registration: hostId, display name, binding equal to the declared one + * B3 resolveStateDir honours an explicit product config path + * P1 config dir: TOKENPILOT_OPENCODE_CONFIG_DIR > $XDG_CONFIG_HOME/opencode > ~/.config/opencode; + * TOKENPILOT_OPENCODE_CONFIG overrides the config file + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { TOKENPILOT_PRESET_ID, TOKENPILOT_PRESET_VERSION } from "@lightrsi/tokenpilot"; + +import { defaultOpenCodeConfigDir, defaultOpenCodeStateDir, defaultTokenPilotOpenCodeConfigPath } from "../src/config.js"; +import { OPENCODE_TOKENPILOT_HOST_BINDING, initializeOpenCodeTokenPilotPreset, openCodeSupportsEviction } from "../src/preset.js"; +import { OPENCODE_PRODUCT_HOST_REGISTRATION } from "../src/product-registration.js"; + +describe("opencode binding and registration", () => { + it("B1 host binding declares the supported subset", () => { + assert.equal(OPENCODE_TOKENPILOT_HOST_BINDING.hostId, "opencode"); + assert.equal(OPENCODE_TOKENPILOT_HOST_BINDING.presetId, TOKENPILOT_PRESET_ID); + assert.equal(OPENCODE_TOKENPILOT_HOST_BINDING.presetVersion, TOKENPILOT_PRESET_VERSION); + assert.deepEqual(OPENCODE_TOKENPILOT_HOST_BINDING.supportedFeatures, ["stabilizer", "reduction", "eviction"]); + assert.equal(openCodeSupportsEviction(), true); + assert.doesNotThrow(() => initializeOpenCodeTokenPilotPreset()); + }); + it("B2 product registration", () => { + assert.equal(OPENCODE_PRODUCT_HOST_REGISTRATION.hostId, "opencode"); + assert.equal(OPENCODE_PRODUCT_HOST_REGISTRATION.displayName, "OpenCode"); + assert.deepEqual(OPENCODE_PRODUCT_HOST_REGISTRATION.preset, OPENCODE_TOKENPILOT_HOST_BINDING); + }); + it("B3 resolveStateDir uses the product config path", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-oc-reg-")); + const configPath = join(dir, "tokenpilot.json"); + await writeFile(configPath, JSON.stringify({ stateDir: join(dir, "custom-state") })); + assert.equal(await OPENCODE_PRODUCT_HOST_REGISTRATION.resolveStateDir({ productConfigPath: configPath }), join(dir, "custom-state")); + }); + it("P1 config dir precedence", () => { + const keys = ["TOKENPILOT_OPENCODE_CONFIG_DIR", "XDG_CONFIG_HOME", "HOME", "TOKENPILOT_OPENCODE_CONFIG"] as const; + const saved = Object.fromEntries(keys.map((k) => [k, process.env[k]])); + try { + for (const k of keys) delete process.env[k]; + process.env.HOME = "/home/u"; + assert.equal(defaultOpenCodeConfigDir(), "/home/u/.config/opencode"); + process.env.XDG_CONFIG_HOME = "/xdg"; + assert.equal(defaultOpenCodeConfigDir(), "/xdg/opencode"); + process.env.TOKENPILOT_OPENCODE_CONFIG_DIR = "/override"; + assert.equal(defaultOpenCodeConfigDir(), "/override"); + assert.equal(defaultTokenPilotOpenCodeConfigPath(), "/override/tokenpilot.json"); + assert.equal(defaultOpenCodeStateDir(), "/override/tokenpilot-state/tokenpilot"); + process.env.TOKENPILOT_OPENCODE_CONFIG = "/cfg/tp.json"; + assert.equal(defaultTokenPilotOpenCodeConfigPath(), "/cfg/tp.json"); + } finally { + for (const k of keys) { if (saved[k] === undefined) delete process.env[k]; else process.env[k] = saved[k]; } + } + }); +}); diff --git a/components/adapters/opencode/tests/codec-overlay.test.ts b/components/adapters/opencode/tests/codec-overlay.test.ts new file mode 100644 index 00000000..35963f77 --- /dev/null +++ b/components/adapters/opencode/tests/codec-overlay.test.ts @@ -0,0 +1,249 @@ +/** + * Regression matrix: opencode/src/codec.ts + opencode/src/overlay.ts + * + * decodeOpenCodeMessages (mirrors toModelMessagesEffect in OpenCode 1.18.33) + * D1 user text parts → text blocks; `ignored` and empty text skipped; image files → image + * blocks; text/plain files and other parts not decoded; empty user → no message + * D2 assistant text → text blocks (empty text kept); reasoning → metadata.reasoning; + * completed/error tool parts → tool_call blocks + one `tool` message per result + * D3 completed tool output → success; natively pruned (time.compacted) → placeholder + * D4 error tool → error status with error text; interrupted error with output → success + * with the partial output + * D5 pending/running tools, step/patch/snapshot/agent/retry/compaction parts → not decoded + * D6 compaction-summary assistant messages carry metadata.ocSummary + * D7 malformed entries (non-object, missing info/parts) are skipped, never thrown + * D8 metadata carries message id / index and, for tool results, the part id + * applyOpenCodeChanges + * A1 no canonical change → 0 replacements, same array elements + * A2 tool output change → cloned part (state.output) in a cloned message; storage + * objects untouched + * A3 error tool change → state.error; interrupted error → metadata.output + * A4 natively pruned part → never written + * A5 user text change → first text part gets the text, other text parts emptied + * A6 assistant text change → same rule; tool parts untouched + * A7 several changes in one message → one replacement + * openCodeSurfaceId / openCodeSurfaceEntries + * S1 tool results are `part:`, everything else `msg:` + * S2 turns increment on user messages + * S3 summaries are checkpoints; overlaid ids and pruned outputs are alreadyEvicted + * overlay store + * O1 missing file → empty overlay; O2 save → load round trip; O3 corrupt file → empty + * O4 applyOverlay: tool results get the stored text; assistant text is replaced while + * tool_call blocks are kept; non-overlaid messages keep identity; count reported + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp, mkdir, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; + +import { + OC_PRUNED_PLACEHOLDER, + applyOpenCodeChanges, + decodeOpenCodeMessages, + openCodeSurfaceEntries, + openCodeSurfaceId, +} from "../src/codec.js"; +import type { OcMessageWithParts, OcPart, OcToolPart } from "../src/opencode-types.js"; +import { applyOverlay, loadOverlay, overlayPath, saveOverlay } from "../src/overlay.js"; + +const S = "ses_1"; +let seq = 0; +const id = (prefix: string) => `${prefix}_${++seq}`; + +function userMsg(parts: OcPart[], messageId = id("msg")): OcMessageWithParts { + return { info: { id: messageId, sessionID: S, role: "user", model: { providerID: "llama", modelID: "qwen3" } }, parts }; +} +function assistantMsg(parts: OcPart[], extra: Record = {}, messageId = id("msg")): OcMessageWithParts { + return { info: { id: messageId, sessionID: S, role: "assistant", ...extra }, parts }; +} +const text = (t: string, extra: Record = {}): OcPart => ({ id: id("prt"), sessionID: S, messageID: "m", type: "text", text: t, ...extra }); +function tool(callID: string, state: OcToolPart["state"], partId = id("prt")): OcToolPart { + return { id: partId, sessionID: S, messageID: "m", type: "tool", callID, tool: "bash", state }; +} +const completed = (output: string, compacted?: number): OcToolPart["state"] => ({ + status: "completed", input: { command: "make" }, output, title: "make", metadata: {}, time: { start: 1, end: 2, ...(compacted ? { compacted } : {}) }, +}); + +describe("decodeOpenCodeMessages", () => { + it("D1 user parts", () => { + const decoded = decodeOpenCodeMessages([ + userMsg([text("hello"), text("hidden", { ignored: true }), text(""), { id: "f1", type: "file", mime: "image/png", url: "data:," }, { id: "f2", type: "file", mime: "text/plain", url: "file://x" }]), + userMsg([text("", {}), { id: "sp", type: "step-start" }]), + ]); + assert.equal(decoded.messages.length, 1); + assert.deepEqual(decoded.messages[0]?.content, [{ type: "text", text: "hello" }, { type: "image", mediaType: "image/png" }]); + assert.deepEqual(decoded.bindings[0], { kind: "user", messageIndex: 0, textPartIndexes: [0] }); + }); + it("D2 assistant text, reasoning and tool parts", () => { + const decoded = decodeOpenCodeMessages([assistantMsg([ + { id: "r", type: "reasoning", text: "think" }, + text(""), + text("Running make."), + tool("call_1", completed("ok")), + ])]); + assert.deepEqual(decoded.messages.map((m) => m.role), ["assistant", "tool"]); + assert.deepEqual(decoded.messages[0]?.content, [ + { type: "text", text: "" }, + { type: "text", text: "Running make." }, + { type: "tool_call", toolCallId: "call_1", toolName: "bash", argumentsJson: { command: "make" } }, + ]); + assert.deepEqual(decoded.messages[0]?.metadata?.reasoning, ["think"]); + assert.deepEqual(decoded.bindings.map((b) => b.kind), ["assistant", "tool"]); + }); + it("D3 completed output and native pruning", () => { + const decoded = decodeOpenCodeMessages([assistantMsg([tool("a", completed("out")), tool("b", completed("old", 99))])]); + const results = decoded.messages.filter((m) => m.role === "tool").map((m) => (m.content as Array<{ text: string; status: string }>)[0]); + assert.deepEqual(results.map((r) => [r?.text, r?.status]), [["out", "success"], [OC_PRUNED_PLACEHOLDER, "success"]]); + }); + it("D4 errors and interrupted errors", () => { + const decoded = decodeOpenCodeMessages([assistantMsg([ + tool("e", { status: "error", input: {}, error: "boom", time: { start: 1, end: 2 } }), + tool("i", { status: "error", input: {}, error: "interrupted", metadata: { interrupted: true, output: "partial" }, time: { start: 1, end: 2 } }), + ])]); + const results = decoded.messages.filter((m) => m.role === "tool").map((m) => (m.content as Array<{ text: string; status: string }>)[0]); + assert.deepEqual(results.map((r) => [r?.text, r?.status]), [["boom", "error"], ["partial", "success"]]); + }); + it("D5 pending/running tools and structural parts are not decoded", () => { + const decoded = decodeOpenCodeMessages([assistantMsg([ + tool("p", { status: "pending", input: {} }), + tool("r", { status: "running", input: {} }), + { id: "s", type: "step-start" }, { id: "f", type: "step-finish" }, { id: "pa", type: "patch" }, { id: "sn", type: "snapshot" }, + ])]); + assert.deepEqual(decoded.messages.map((m) => [m.role, m.content]), [["assistant", []]]); + }); + it("D6 compaction summaries are flagged", () => { + const decoded = decodeOpenCodeMessages([assistantMsg([text("summary")], { summary: true })]); + assert.equal(decoded.messages[0]?.metadata?.ocSummary, true); + }); + it("D7 malformed entries are skipped", () => { + const decoded = decodeOpenCodeMessages([null, { info: null }, { info: { role: "user" } }, { parts: [] }] as unknown as OcMessageWithParts[]); + assert.deepEqual(decoded.messages, []); + }); + it("D8 metadata identifies the source message and part", () => { + const part = tool("c", completed("x"), "prt_fixed"); + const decoded = decodeOpenCodeMessages([assistantMsg([part], {}, "msg_fixed")]); + assert.deepEqual(decoded.messages[1]?.metadata, { ocMessageId: "msg_fixed", ocIndex: 0, ocPartId: "prt_fixed" }); + }); +}); + +describe("applyOpenCodeChanges", () => { + function fixture() { + const messages: OcMessageWithParts[] = [ + userMsg([text("fix it"), text("please")]), + assistantMsg([text("On it."), tool("c1", completed("x".repeat(100))), tool("c2", { status: "error", input: {}, error: "boom", time: { start: 1, end: 2 } }), tool("c3", completed("old", 5)), tool("c4", { status: "error", input: {}, error: "int", metadata: { interrupted: true, output: "partial" }, time: { start: 1, end: 2 } })]), + ]; + const snapshot = JSON.stringify(messages); + const originals = [...messages]; + const decoded = decodeOpenCodeMessages(messages); + const canonical = decoded.messages.map((m) => ({ ...m, content: Array.isArray(m.content) ? m.content.map((b) => ({ ...b })) : m.content })); + return { messages, snapshot, originals, decoded, canonical }; + } + const setText = (canonical: ReturnType["canonical"], index: number, value: string) => { + const content = canonical[index]!.content as Array<{ type: string; text: string }>; + const block = content.find((b) => b.type === "text" || b.type === "tool_result")!; + block.text = value; + }; + it("A1 no change leaves every element as is", () => { + const f = fixture(); + assert.equal(applyOpenCodeChanges(f.messages, f.decoded, f.canonical), 0); + f.messages.forEach((m, i) => assert.equal(m, f.originals[i])); + }); + it("A2 a tool output change clones the part and message", () => { + const f = fixture(); + setText(f.canonical, 2, "trimmed"); + assert.equal(applyOpenCodeChanges(f.messages, f.decoded, f.canonical), 1); + assert.notEqual(f.messages[1], f.originals[1]); + assert.equal((f.messages[1]!.parts[1] as OcToolPart).state.status, "completed"); + assert.equal(((f.messages[1]!.parts[1] as OcToolPart).state as { output: string }).output, "trimmed"); + assert.equal(JSON.stringify(f.originals), f.snapshot, "stored objects mutated"); + }); + it("A3 error and interrupted-error changes", () => { + const f = fixture(); + setText(f.canonical, 3, "boom-short"); + setText(f.canonical, 5, "partial-short"); + applyOpenCodeChanges(f.messages, f.decoded, f.canonical); + assert.equal(((f.messages[1]!.parts[2] as OcToolPart).state as { error: string }).error, "boom-short"); + const interrupted = (f.messages[1]!.parts[4] as OcToolPart).state as unknown as { error: string; metadata: { output: string } }; + assert.deepEqual([interrupted.error, interrupted.metadata.output], ["int", "partial-short"]); + }); + it("A4 natively pruned parts are never written", () => { + const f = fixture(); + setText(f.canonical, 4, "anything"); + assert.equal(applyOpenCodeChanges(f.messages, f.decoded, f.canonical), 0); + }); + it("A5 user text change fills the first part and empties the rest", () => { + const f = fixture(); + (f.canonical[0]!.content as Array<{ text: string }>)[0]!.text = "DATE\n\nfix it"; + applyOpenCodeChanges(f.messages, f.decoded, f.canonical); + assert.deepEqual(f.messages[0]!.parts.map((p) => (p as { text: string }).text), ["DATE\n\nfix it\nplease", ""]); + }); + it("A6 assistant text change keeps tool parts", () => { + const f = fixture(); + setText(f.canonical, 1, "[evicted: message x]"); + applyOpenCodeChanges(f.messages, f.decoded, f.canonical); + assert.equal((f.messages[1]!.parts[0] as { text: string }).text, "[evicted: message x]"); + f.messages[1]!.parts.slice(1).forEach((part, i) => assert.equal(part, f.originals[1]!.parts[i + 1])); + }); + it("A7 several changes in one message produce one replacement", () => { + const f = fixture(); + setText(f.canonical, 1, "short"); + setText(f.canonical, 2, "trimmed"); + assert.equal(applyOpenCodeChanges(f.messages, f.decoded, f.canonical), 1); + }); +}); + +describe("surface ids and entries", () => { + const messages = [ + userMsg([text("one")], "msg_u1"), + assistantMsg([text("done"), tool("c1", completed("o"), "prt_t1")], {}, "msg_a1"), + assistantMsg([text("summary")], { summary: true }, "msg_sum"), + userMsg([text("two")], "msg_u2"), + assistantMsg([tool("c2", completed("old", 9), "prt_t2")], {}, "msg_a2"), + ]; + const decoded = decodeOpenCodeMessages(messages); + it("S1 ids", () => { + assert.deepEqual(decoded.messages.map(openCodeSurfaceId), ["msg:msg_u1", "msg:msg_a1", "part:prt_t1", "msg:msg_sum", "msg:msg_u2", "msg:msg_a2", "part:prt_t2"]); + }); + it("S2 turns", () => { + assert.deepEqual(openCodeSurfaceEntries(decoded.messages, new Set()).map((e) => e.turn), [1, 1, 1, 1, 2, 2, 2]); + }); + it("S3 checkpoints and already-evicted entries", () => { + const entries = openCodeSurfaceEntries(decoded.messages, new Set(["part:prt_t1"])); + assert.deepEqual(entries.filter((e) => e.checkpoint).map((e) => e.id), ["msg:msg_sum"]); + assert.deepEqual(entries.filter((e) => e.alreadyEvicted).map((e) => e.id), ["part:prt_t1", "part:prt_t2"]); + }); +}); + +describe("overlay store", () => { + it("O1 a missing overlay is empty", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-oc-ov-")); + assert.deepEqual(await loadOverlay(dir, S), { version: 1, sessionId: S, entries: {} }); + }); + it("O2 save and load round trip", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-oc-ov-")); + const overlay = { version: 1 as const, sessionId: S, entries: { "part:p": { text: "stub", at: "t" } } }; + await saveOverlay(dir, overlay); + assert.deepEqual(await loadOverlay(dir, S), overlay); + }); + it("O3 a corrupt overlay is treated as empty", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-oc-ov-")); + const path = overlayPath(dir, S); + await mkdir(dirname(path), { recursive: true }); + await writeFile(path, "{broken"); + assert.deepEqual((await loadOverlay(dir, S)).entries, {}); + }); + it("O4 applyOverlay replaces overlaid text and keeps everything else", () => { + const decoded = decodeOpenCodeMessages([ + userMsg([text("one")], "msg_u1"), + assistantMsg([text("long answer"), tool("c1", completed("big"), "prt_t1")], {}, "msg_a1"), + ]); + const { messages, applied } = applyOverlay(decoded.messages, { + version: 1, sessionId: S, entries: { "part:prt_t1": { text: "[evicted: tool_result part:prt_t1]", at: "t" }, "msg:msg_a1": { text: "[evicted: message]", at: "t" } }, + }); + assert.equal(applied, 2); + assert.equal(messages[0], decoded.messages[0]); + assert.deepEqual(messages[1]?.content, [{ type: "text", text: "[evicted: message]" }, { type: "tool_call", toolCallId: "c1", toolName: "bash", argumentsJson: { command: "make" } }]); + assert.equal((messages[2]?.content as Array<{ text: string }>)[0]?.text, "[evicted: tool_result part:prt_t1]"); + }); +}); diff --git a/components/adapters/opencode/tests/install-doctor.test.ts b/components/adapters/opencode/tests/install-doctor.test.ts new file mode 100644 index 00000000..6b7bcb9b --- /dev/null +++ b/components/adapters/opencode/tests/install-doctor.test.ts @@ -0,0 +1,306 @@ +/** + * Regression matrix: opencode/src/install.ts + opencode/src/doctor.ts + * + * installOpenCodeTokenPilot — plugin loader + * I1 missing bundle → throws, nothing written + * I2 fresh config dir → marker ESM loader re-exporting the bundle URL; normal-mode + * tokenpilot.json; state dir; manifest; CLI context remembers both config paths + * I3 foreign file at the loader path → timestamped backup, recorded + * I4 existing tokenpilot.json kept byte-identical; no credentials ever written + * installOpenCodeTokenPilot — recovery MCP registration (partitioned by opencode config state) + * J1 no opencode.json / .jsonc → opencode.json created with $schema + mcp entry + * J2 existing opencode.json → backup written, other keys and other MCP servers kept + * J3 existing entry → replaced (reason "updated") + * J4 only opencode.jsonc → nothing written, reason jsonc_only, snippet returned + * J5 unparseable opencode.json → nothing written, reason unparseable + * J6 entry shape: local, [node, server.js], TOKENPILOT_STATE_DIR = state dir, enabled + * renderOpenCodeLoader / openCodeLoaderTarget / isTokenPilotOpenCodeLoader + * L1 round trip loader → bundle path; L2 marker detection + * uninstallOpenCodeTokenPilot + * U1 removes loader and only our MCP key; other keys and servers kept + * U2 opencode.json created by install and now empty → removed + * U3 loader backup restored; foreign loader kept + * U4 --purge removes created tokenpilot.json + state; keeps a pre-existing one + * U5 uninstall without install → no-op + * inspectOpenCodeDoctor + * D1 fresh install → healthy, MCP matches, probe ok + * D2 nothing installed → config, loader and MCP problems + * D3 MCP entry pointing at another state dir → mismatch problem + * D4 jsonc-only config → problem pointing at the snippet + * D5 native pruning on → warning; off by default + * D6 user target → warning + * D7 formatted report lists every check + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync } from "node:fs"; +import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { loadTokenPilotOpenCodeConfig, normalizeTokenPilotOpenCodeConfig } from "../src/config.js"; +import { formatOpenCodeDoctorReport, inspectOpenCodeDoctor, openCodeLoaderTarget } from "../src/doctor.js"; +import { + OPENCODE_LOADER_MARKER, + OPENCODE_MCP_KEY, + installOpenCodeTokenPilot, + isTokenPilotOpenCodeLoader, + renderOpenCodeLoader, + uninstallOpenCodeTokenPilot, +} from "../src/install.js"; + +async function fixture() { + const root = await mkdtemp(join(tmpdir(), "tp-oc-install-")); + const configDir = join(root, "opencode"); + const bundlePath = join(root, "bundle", "plugin.mjs"); + await mkdir(join(root, "bundle"), { recursive: true }); + await writeFile(bundlePath, "export default { id: 'tokenpilot', server: async () => ({}) };\n"); + const cliContextPath = join(root, "cli-context.json"); + return { + root, configDir, bundlePath, cliContextPath, + loaderPath: join(configDir, "plugins", "tokenpilot.js"), + jsonPath: join(configDir, "opencode.json"), + jsoncPath: join(configDir, "opencode.jsonc"), + tpConfigPath: join(configDir, "tokenpilot.json"), + install: (extra: Record = {}) => installOpenCodeTokenPilot({ configDir, bundlePath, cliContextPath, installCliBin: false, ...extra }), + uninstall: (extra: Record = {}) => uninstallOpenCodeTokenPilot({ configDir, ...extra }), + }; +} + +const readJson = async (path: string) => JSON.parse(await readFile(path, "utf8")); + +describe("install: plugin loader", () => { + it("I1 a missing bundle throws before writing", async () => { + const f = await fixture(); + await assert.rejects(f.install({ bundlePath: join(f.root, "none.mjs") }), /not built/); + assert.equal(existsSync(f.configDir), false); + }); + it("I2 a fresh install writes loader, config, state, manifest and CLI context", async () => { + const f = await fixture(); + const result = await f.install(); + assert.equal(await readFile(f.loaderPath, "utf8"), renderOpenCodeLoader(f.bundlePath)); + assert.equal(result.tokenPilotConfigCreated, true); + assert.deepEqual(await loadTokenPilotOpenCodeConfig(f.tpConfigPath), normalizeTokenPilotOpenCodeConfig({}, { configPath: f.tpConfigPath })); + assert.ok(existsSync(result.stateDir)); + assert.ok(existsSync(join(f.configDir, "tokenpilot-install.json"))); + const context = await readJson(f.cliContextPath); + assert.deepEqual(context.configPathsByHost.opencode, { tokenPilotConfigPath: f.tpConfigPath, hostConfigPath: f.jsonPath }); + }); + it("I3 a foreign plugin file is backed up", async () => { + const f = await fixture(); + await mkdir(join(f.configDir, "plugins"), { recursive: true }); + await writeFile(f.loaderPath, "export default {};\n"); + const result = await f.install({ now: () => new Date(1_700_000_000_000) }); + assert.equal(result.loaderBackupPath, `${f.loaderPath}.bak-1700000000000`); + assert.equal(await readFile(result.loaderBackupPath!, "utf8"), "export default {};\n"); + }); + it("I4 an existing tokenpilot.json is kept and no credentials are written", async () => { + const f = await fixture(); + await mkdir(f.configDir, { recursive: true }); + const custom = `${JSON.stringify({ reduction: { maxToolChars: 1800 } })}\n`; + await writeFile(f.tpConfigPath, custom); + const result = await f.install(); + assert.equal(result.tokenPilotConfigCreated, false); + assert.equal(await readFile(f.tpConfigPath, "utf8"), custom); + const fresh = await fixture(); + await fresh.install(); + assert.equal((await readJson(fresh.tpConfigPath)).taskStateEstimator.apiKey, undefined); + }); +}); + +describe("install: recovery MCP registration", () => { + it("J1 creates opencode.json when none exists", async () => { + const f = await fixture(); + const result = await f.install(); + assert.deepEqual([result.mcp.registered, result.mcp.reason, result.mcp.opencodeConfigCreated], [true, "registered", true]); + const root = await readJson(f.jsonPath); + assert.equal(root.$schema, "https://opencode.ai/config.json"); + assert.ok(root.mcp[OPENCODE_MCP_KEY]); + }); + it("J2 updates an existing opencode.json with a backup, keeping other settings", async () => { + const f = await fixture(); + await mkdir(f.configDir, { recursive: true }); + const original = { model: "llama/qwen3", mcp: { other: { type: "remote", url: "https://x" } } }; + await writeFile(f.jsonPath, JSON.stringify(original)); + const result = await f.install({ now: () => new Date(1_700_000_000_000) }); + assert.equal(result.mcp.backupPath, `${f.jsonPath}.tokenpilot-backup-1700000000000`); + assert.deepEqual(await readJson(result.mcp.backupPath!), original); + const root = await readJson(f.jsonPath); + assert.equal(root.model, "llama/qwen3"); + assert.deepEqual(root.mcp.other, original.mcp.other); + assert.ok(root.mcp[OPENCODE_MCP_KEY]); + }); + it("J3 replaces an existing TokenPilot entry", async () => { + const f = await fixture(); + await mkdir(f.configDir, { recursive: true }); + await writeFile(f.jsonPath, JSON.stringify({ mcp: { [OPENCODE_MCP_KEY]: { type: "local", command: ["old"] } } })); + const result = await f.install(); + assert.equal(result.mcp.reason, "updated"); + assert.notDeepEqual((await readJson(f.jsonPath)).mcp[OPENCODE_MCP_KEY].command, ["old"]); + }); + it("J4 never rewrites an opencode.jsonc-only setup", async () => { + const f = await fixture(); + await mkdir(f.configDir, { recursive: true }); + await writeFile(f.jsoncPath, "{ // my comments\n}\n"); + const result = await f.install(); + assert.deepEqual([result.mcp.registered, result.mcp.reason], [false, "jsonc_only"]); + assert.equal(existsSync(f.jsonPath), false); + assert.equal(await readFile(f.jsoncPath, "utf8"), "{ // my comments\n}\n"); + assert.ok(result.mcp.snippet.includes(OPENCODE_MCP_KEY)); + }); + it("J5 leaves an unparseable opencode.json untouched", async () => { + const f = await fixture(); + await mkdir(f.configDir, { recursive: true }); + await writeFile(f.jsonPath, "{ nope"); + const result = await f.install(); + assert.deepEqual([result.mcp.registered, result.mcp.reason], [false, "unparseable"]); + assert.equal(await readFile(f.jsonPath, "utf8"), "{ nope"); + }); + it("J6 the entry runs the shared recovery server against the state dir", async () => { + const f = await fixture(); + const result = await f.install(); + const entry = (await readJson(f.jsonPath)).mcp[OPENCODE_MCP_KEY]; + assert.equal(entry.type, "local"); + assert.equal(entry.command[0], process.execPath); + assert.match(entry.command[1], /products[\\/]mcp[\\/]dist[\\/]server\.js$/); + assert.deepEqual(entry.environment, { TOKENPILOT_STATE_DIR: result.stateDir }); + assert.equal(entry.enabled, true); + }); +}); + +describe("loader helpers", () => { + it("L1 the loader target round-trips", async () => { + const f = await fixture(); + await f.install(); + assert.equal(openCodeLoaderTarget(f.loaderPath), f.bundlePath); + await writeFile(f.loaderPath, "garbage"); + assert.equal(openCodeLoaderTarget(f.loaderPath), undefined); + }); + it("L2 marker detection", async () => { + const f = await fixture(); + assert.equal(await isTokenPilotOpenCodeLoader(f.loaderPath), false); + await f.install(); + assert.equal(await isTokenPilotOpenCodeLoader(f.loaderPath), true); + assert.ok(renderOpenCodeLoader("/x").startsWith(OPENCODE_LOADER_MARKER)); + }); +}); + +describe("uninstallOpenCodeTokenPilot", () => { + it("U1 removes the loader and only our MCP key", async () => { + const f = await fixture(); + await mkdir(f.configDir, { recursive: true }); + await writeFile(f.jsonPath, JSON.stringify({ model: "m", mcp: { other: { type: "remote", url: "https://x" } } })); + await f.install(); + const result = await f.uninstall(); + assert.deepEqual([result.loaderRemoved, result.mcpRemoved, result.opencodeConfigRemoved], [true, true, false]); + assert.deepEqual(await readJson(f.jsonPath), { model: "m", mcp: { other: { type: "remote", url: "https://x" } } }); + }); + it("U2 an opencode.json created by install and now empty is removed", async () => { + const f = await fixture(); + await f.install(); + const result = await f.uninstall(); + assert.equal(result.opencodeConfigRemoved, true); + assert.equal(existsSync(f.jsonPath), false); + }); + it("U3 restores a loader backup and keeps a foreign loader", async () => { + const f = await fixture(); + await mkdir(join(f.configDir, "plugins"), { recursive: true }); + await writeFile(f.loaderPath, "mine\n"); + await f.install(); + const restored = await f.uninstall(); + assert.equal(restored.loaderBackupRestored, true); + assert.equal(await readFile(f.loaderPath, "utf8"), "mine\n"); + const g = await fixture(); + await g.install(); + await writeFile(g.loaderPath, "someone else\n"); + assert.equal((await g.uninstall()).foreignLoaderKept, true); + }); + it("U4 --purge removes created config and state, keeps a pre-existing config", async () => { + const f = await fixture(); + const installed = await f.install(); + const purged = await f.uninstall({ purge: true }); + assert.deepEqual([purged.configRemoved, purged.stateRemoved], [true, true]); + assert.equal(existsSync(installed.stateDir), false); + assert.equal(existsSync(join(f.configDir, "tokenpilot-state")), false); + const g = await fixture(); + await mkdir(g.configDir, { recursive: true }); + await writeFile(g.tpConfigPath, "{}\n"); + await g.install(); + assert.equal((await g.uninstall({ purge: true })).configRemoved, false); + assert.ok(existsSync(g.tpConfigPath)); + }); + it("U5 uninstall without an install is a no-op", async () => { + const f = await fixture(); + assert.deepEqual(await f.uninstall(), { + loaderRemoved: false, loaderBackupRestored: false, foreignLoaderKept: false, mcpRemoved: false, + opencodeConfigRemoved: false, configRemoved: false, stateRemoved: false, + }); + }); +}); + +describe("doctor", () => { + async function installed() { + const f = await fixture(); + await f.install(); + const config = await loadTokenPilotOpenCodeConfig(f.tpConfigPath); + const inspect = (cfg = config, probeMcp = false) => inspectOpenCodeDoctor({ config: cfg, configPath: f.tpConfigPath, configDir: f.configDir, detectVersion: false, probeMcp }); + return { ...f, config, inspect }; + } + it("D1 a fresh install is healthy and the MCP server answers", async () => { + const f = await installed(); + const report = await f.inspect(f.config, true); + assert.deepEqual(report.problems, []); + assert.equal(report.mcpMatchesExpected, true); + assert.equal(report.mcpProbe?.ok, true); + assert.equal(report.nativePrune, "off"); + assert.deepEqual(report.declaredFeatures, ["stabilizer", "reduction", "eviction"]); + }); + it("D2 nothing installed is unhealthy", async () => { + const f = await fixture(); + const report = await inspectOpenCodeDoctor({ + config: normalizeTokenPilotOpenCodeConfig({}, { configPath: f.tpConfigPath }), configPath: f.tpConfigPath, configDir: f.configDir, detectVersion: false, probeMcp: false, + }); + for (const prefix of ["config missing", "plugin loader missing", "recovery MCP not registered"]) { + assert.ok(report.problems.some((p) => p.startsWith(prefix)), prefix); + } + }); + it("D3 an MCP entry for another state dir is a mismatch", async () => { + const f = await installed(); + const root = await readJson(f.jsonPath); + root.mcp[OPENCODE_MCP_KEY].environment.TOKENPILOT_STATE_DIR = "/elsewhere"; + await writeFile(f.jsonPath, JSON.stringify(root)); + const report = await f.inspect(); + assert.equal(report.mcpMatchesExpected, false); + assert.ok(report.problems.some((p) => p.startsWith("recovery MCP entry does not match"))); + }); + it("D4 a jsonc-only setup points at the snippet", async () => { + const f = await fixture(); + await mkdir(f.configDir, { recursive: true }); + await writeFile(f.jsoncPath, "{}\n"); + await f.install(); + const report = await inspectOpenCodeDoctor({ config: await loadTokenPilotOpenCodeConfig(f.tpConfigPath), configPath: f.tpConfigPath, configDir: f.configDir, detectVersion: false, probeMcp: false }); + assert.equal(report.opencodeConfigState, "jsonc_only"); + assert.ok(report.problems.some((p) => p.includes("paste the snippet"))); + }); + it("D5 native pruning is reported when on", async () => { + const f = await installed(); + const root = await readJson(f.jsonPath); + await writeFile(f.jsonPath, JSON.stringify({ ...root, compaction: { prune: true } })); + const report = await f.inspect(); + assert.equal(report.nativePrune, "on"); + assert.ok(report.warnings.some((w) => w.includes("native pruning"))); + }); + it("D6 user target is reported as unsupported", async () => { + const f = await installed(); + const report = await f.inspect({ ...f.config, hooks: { dynamicContextTarget: "user" } }); + assert.ok(report.warnings.some((w) => w.startsWith("dynamicContextTarget=user is not supported"))); + assert.equal(report.healthy, true); + }); + it("D7 the formatted report lists every check", async () => { + const f = await installed(); + const text = formatOpenCodeDoctorReport(await f.inspect()); + for (const label of ["tokenpilot config", "enabled", "opencode version", "plugin loader", "plugin bundle", "opencode config", "recovery MCP registered", "recovery MCP server built", "recovery MCP probe", "native tool-output pruning", "stateDir", "declared features", "stabilizer", "reduction", "recovery protocol injected", "eviction", "latest session", "latest activity", "adapter log", "healthy: yes"]) { + assert.ok(text.includes(`- ${label}`), label); + } + }); +}); diff --git a/components/adapters/opencode/tests/runtime.test.ts b/components/adapters/opencode/tests/runtime.test.ts new file mode 100644 index 00000000..9f62606f --- /dev/null +++ b/components/adapters/opencode/tests/runtime.test.ts @@ -0,0 +1,353 @@ +/** + * Regression matrix: opencode/src/runtime.ts + opencode/src/plugin.ts + * + * Plugin module + * M1 default export is exactly { id: "tokenpilot", server }, no other exports + * M2 server() returns the two transform hooks and dispose + * Config loading (ensureConfig) + * G1 disabled config → both hooks leave output untouched + * G2 config file edits are picked up on the next hook call (mtime/size stamp) + * G3 loader throws → hooks fail open (output untouched), warning logged + * experimental.chat.system.transform + * Y1 volatile line moved to the tail (shared stabilizer, developer) + recovery protocol + * Y2 no volatile lines → recovery protocol appended only + * Y3 idempotent: running twice does not duplicate the protocol + * Y4 dynamicContextTarget=user → developer behaviour, one warning logged + * Y5 stabilizer off → volatile line stays; protocol still appended + * Y6 malformed output (no system array / non-string) → untouched, no throw + * experimental.chat.messages.transform + * R1 large tool output → only that part is replaced (cloned); text equals the shared + * reduction run directly; latest-session ref and a turn binding (total + per-pass + * savings) are written + * R2 repeated call on a fresh copy of the same history → byte-identical output (memo) + * R3 small history → no replacements (array elements keep identity) + * R4 no session id → untouched + * R5 malformed messages → untouched, no throw + * R6 reduction module off → no replacements + * R7 a trimmed file read stays byte-identical on the next request after another + * tool turn is appended (regression: repeat-read detection must not fire on the + * same read resent with the full history) + * R8 a restarted OpenCode process (new runtime, same stateDir + session) sends the + * byte-identical history (archive paths are content-derived) + * R9 disclosed-read memo save failure → the reduction is still applied, warning logged + * Eviction overlay (messages.transform) + * E1 default config → estimator never created, no overlay written + * E2 enabled + completed earlier turn → overlay saved and applied (stub text in the part) + * E3 a later request re-applies the stored overlay without calling the estimator + * E4 overlay save failure → no eviction applied this request (deferred) + * E5 estimator throws → eviction skipped and logged, reduction still applied + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { chmod, mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; + +import type { SemanticTaskUpdate, TaskStateEstimator } from "@lightrsi/eviction"; +import { createEmptySessionTaskRegistry, type SessionTaskRegistry } from "@lightrsi/history"; +import { latestSessionPath, loadRecentTurnBindings } from "@lightrsi/host-adapter"; + +import { canonicalRecoveryProtocolText, createCanonicalEnvelope } from "../../shared/canonical/before-call.js"; +import { adapterLogPath } from "../../shared/canonical/logger.js"; +import { reduceCanonicalEnvelope } from "../../shared/canonical/reduction.js"; +import { decodeOpenCodeMessages } from "../src/codec.js"; +import { normalizeTokenPilotOpenCodeConfig } from "../src/config.js"; +import type { OcMessageWithParts, OcToolPart } from "../src/opencode-types.js"; +import { overlayPath } from "../src/overlay.js"; +import * as pluginModule from "../src/plugin.js"; +import { OpenCodeTokenPilotRuntime, type OpenCodeRuntimeDependencies } from "../src/runtime.js"; + +const S = "ses_rt"; +const BIG = Array.from({ length: 3000 }, (_, i) => `line ${i} output from build step ok`).join("\n"); +const SYSTEM = "You are opencode.\nToday's date: Mon Sep 28 2026\nWorking directory: /work"; + +async function setup(raw: Record = {}, deps: OpenCodeRuntimeDependencies = {}) { + const dir = await mkdtemp(join(tmpdir(), "tp-oc-rt-")); + const configPath = join(dir, "tokenpilot.json"); + await writeFile(configPath, JSON.stringify({ stateDir: join(dir, "state"), ...raw })); + const runtime = new OpenCodeTokenPilotRuntime({ configPath, ...deps }); + const hooks = runtime.hooks(); + const stateDir = join(dir, "state"); + return { dir, configPath, stateDir, runtime, hooks }; +} + +function history(output = BIG, suffix = ""): OcMessageWithParts[] { + return [ + { info: { id: `msg_u1${suffix}`, sessionID: S, role: "user", model: { providerID: "llama", modelID: "qwen3" } }, parts: [{ id: "p_u1", sessionID: S, messageID: "msg_u1", type: "text", text: "fix the build" }] }, + { + info: { id: "msg_a1", sessionID: S, role: "assistant" }, + parts: [ + { id: "p_t1", sessionID: S, messageID: "msg_a1", type: "tool", callID: "call_1", tool: "bash", state: { status: "completed", input: { command: "make" }, output, title: "make", metadata: {}, time: { start: 1, end: 2 } } }, + ], + }, + ]; +} + +const system = (text = SYSTEM) => ({ system: [text] }); + +describe("plugin module", () => { + it("M1 default export only, shaped { id, server }", () => { + assert.deepEqual(Object.keys(pluginModule), ["default"]); + assert.equal(pluginModule.default.id, "tokenpilot"); + assert.equal(typeof pluginModule.default.server, "function"); + }); + it("M2 server returns the transform hooks", async () => { + const hooks = await pluginModule.default.server({ directory: "/w", worktree: "/w" }); + assert.deepEqual(Object.keys(hooks).sort(), ["dispose", "experimental.chat.messages.transform", "experimental.chat.system.transform"]); + }); +}); + +describe("config loading", () => { + it("G1 a disabled config leaves both outputs untouched", async () => { + const env = await setup({ enabled: false }); + const sys = system(); + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, sys); + assert.equal(sys.system[0], SYSTEM); + const messages = history(); + const before = [...messages]; + await env.hooks["experimental.chat.messages.transform"]!({}, { messages }); + messages.forEach((m, i) => assert.equal(m, before[i])); + }); + it("G2 config edits are picked up on the next call", async () => { + const env = await setup({ enabled: false }); + const first = system(); + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, first); + assert.equal(first.system[0], SYSTEM); + await new Promise((resolve) => setTimeout(resolve, 20)); + await writeFile(env.configPath, JSON.stringify({ stateDir: env.stateDir, enabled: true, logLevel: "debug" })); + const second = system(); + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, second); + assert.notEqual(second.system[0], SYSTEM); + }); + it("G3 a throwing loader fails open", async () => { + const env = await setup({}, { loadConfig: async () => { throw new Error("broken config"); } }); + const sys = system(); + await assert.doesNotReject(env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, sys)); + assert.equal(sys.system[0], SYSTEM); + }); +}); + +describe("experimental.chat.system.transform", () => { + it("Y1 moves the volatile line to the tail and appends the recovery protocol", async () => { + const env = await setup(); + const sys = system(); + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: { id: "qwen3" } }, sys); + assert.equal(sys.system[0], `You are opencode.\nWorking directory: /work\n\nToday's date: Mon Sep 28 2026\n\n${canonicalRecoveryProtocolText()}`); + assert.equal(sys.system.length, 1); + }); + it("Y2 without volatile lines only the protocol is appended", async () => { + const env = await setup(); + const sys = system("You are opencode."); + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, sys); + assert.equal(sys.system[0], `You are opencode.\n\n${canonicalRecoveryProtocolText()}`); + }); + it("Y3 is idempotent", async () => { + const env = await setup(); + const sys = system("You are opencode."); + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, sys); + const once = sys.system[0]; + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, sys); + assert.equal(sys.system[0], once); + }); + it("Y4 user target falls back to developer with one warning", async () => { + const env = await setup({ hooks: { dynamicContextTarget: "user" } }); + for (let i = 0; i < 2; i += 1) { + const sys = system(); + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, sys); + assert.ok(sys.system[0]!.includes("\n\nToday's date: Mon Sep 28 2026\n\n")); + } + const log = await readFile(adapterLogPath(env.stateDir), "utf8"); + assert.equal(log.match(/dynamicContextTarget=user is not supported/g)?.length, 1); + }); + it("Y5 stabilizer off keeps the volatile line in place", async () => { + const env = await setup({ modules: { stabilizer: false } }); + const sys = system(); + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, sys); + assert.equal(sys.system[0], `${SYSTEM}\n\n${canonicalRecoveryProtocolText()}`); + }); + it("Y6 malformed output is left alone", async () => { + const env = await setup(); + const empty = { system: [] as string[] }; + await env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, empty); + assert.deepEqual(empty.system, []); + await assert.doesNotReject(env.hooks["experimental.chat.system.transform"]!({ sessionID: S, model: {} }, null as unknown as { system: string[] })); + }); +}); + +describe("experimental.chat.messages.transform", () => { + it("R1 reduces the large output exactly as the shared pipeline would", async () => { + const env = await setup(); + const messages = history(); + const original = [...messages]; + await env.hooks["experimental.chat.messages.transform"]!({}, { messages }); + assert.equal(messages[0], original[0]); + assert.notEqual(messages[1], original[1]); + const output = ((messages[1]!.parts[0] as OcToolPart).state as { output: string }).output; + const direct = await reduceCanonicalEnvelope({ + envelope: createCanonicalEnvelope({ hostId: "opencode", displayName: "OpenCode", sessionId: S, model: "qwen3", messages: decodeOpenCodeMessages(history()).messages }), + config: normalizeTokenPilotOpenCodeConfig({ stateDir: env.stateDir }), + }); + const directText = (direct.envelope.messages[2]!.content as Array<{ text: string }>)[0]!.text; + const strip = (t: string) => t.replace(/\nArchive: [^\n]*$/, ""); + assert.equal(strip(output), strip(directText)); + assert.equal(((original[1]!.parts[0] as OcToolPart).state as { output: string }).output, BIG, "stored part mutated"); + assert.equal(JSON.parse(await readFile(latestSessionPath(env.stateDir), "utf8")).sessionId, S); + const bindings = await loadRecentTurnBindings<{ reductionSavedChars: number; reductionPassSavedChars: Record }>(env.stateDir, S); + assert.equal(bindings.at(-1)?.reductionSavedChars, BIG.length - output.length); + assert.ok((bindings.at(-1)?.reductionPassSavedChars.tool_payload_trim ?? 0) > 0); + }); + it("R2 a repeated call is byte-identical", async () => { + const env = await setup(); + const first = history(); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages: first }); + await new Promise((resolve) => setTimeout(resolve, 5)); + const second = history(); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages: second }); + assert.equal(JSON.stringify(second), JSON.stringify(first)); + }); + it("R3 a small history keeps every element", async () => { + const env = await setup(); + const messages = history("small"); + const before = [...messages]; + await env.hooks["experimental.chat.messages.transform"]!({}, { messages }); + messages.forEach((m, i) => assert.equal(m, before[i])); + }); + it("R4 no session id leaves messages alone", async () => { + const env = await setup(); + const messages = history().map((m) => ({ ...m, info: { ...m.info, sessionID: "" } })); + const before = [...messages]; + await env.hooks["experimental.chat.messages.transform"]!({}, { messages }); + messages.forEach((m, i) => assert.equal(m, before[i])); + }); + it("R5 malformed messages never throw", async () => { + const env = await setup(); + await assert.doesNotReject(env.hooks["experimental.chat.messages.transform"]!({}, { messages: [null, 1] as unknown as OcMessageWithParts[] })); + await assert.doesNotReject(env.hooks["experimental.chat.messages.transform"]!({}, null as unknown as { messages: OcMessageWithParts[] })); + }); + it("R6 reduction off makes no replacements", async () => { + const env = await setup({ modules: { reduction: false } }); + const messages = history(); + const before = [...messages]; + await env.hooks["experimental.chat.messages.transform"]!({}, { messages }); + messages.forEach((m, i) => assert.equal(m, before[i])); + }); + it("R7 a trimmed file read stays byte-identical on the next request", async () => { + const env = await setup(); + const code = Array.from({ length: 400 }, (_, i) => `export function handler${i}(input: string): string {\n return input.trim() + "${i}";\n}\n`).join("\n"); + const toolMessage = (id: string, callID: string, tool: string, input: Record, output: string): OcMessageWithParts => ({ + info: { id, sessionID: S, role: "assistant" }, + parts: [{ id: `p_${id}`, sessionID: S, messageID: id, type: "tool", callID, tool, state: { status: "completed", input, output, title: tool, metadata: {}, time: { start: 1, end: 2 } } }], + }); + const base = (): OcMessageWithParts[] => [ + { info: { id: "msg_u1", sessionID: S, role: "user", model: { providerID: "llama", modelID: "qwen3" } }, parts: [{ id: "p_u1", sessionID: S, messageID: "msg_u1", type: "text", text: "read the handlers" }] }, + toolMessage("msg_a1", "call_r", "read", { filePath: "/work/src/handlers.ts" }, code), + ]; + const first = base(); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages: first }); + const readPart = (messages: OcMessageWithParts[]) => JSON.stringify(messages[1]!.parts[0]); + assert.ok(readPart(first).length < JSON.stringify(base()[1]!.parts[0]).length, "precondition: the read is trimmed"); + const second = [...base(), toolMessage("msg_a2", "call_w", "write", { filePath: "/work/out.txt", content: "x" }, "wrote 1 byte")]; + await env.hooks["experimental.chat.messages.transform"]!({}, { messages: second }); + assert.equal(readPart(second), readPart(first)); + }); + it("R8 a restarted OpenCode process sends byte-identical history", async () => { + const shared = await mkdtemp(join(tmpdir(), "tp-oc-restart-")); + const before = await setup({ stateDir: shared }); + const first = history(); + await before.hooks["experimental.chat.messages.transform"]!({}, { messages: first }); + assert.ok(JSON.stringify(first).length < JSON.stringify(history()).length, "precondition: reduced"); + await new Promise((resolve) => setTimeout(resolve, 5)); + const after = await setup({ stateDir: shared }); + const second = history(); + await after.hooks["experimental.chat.messages.transform"]!({}, { messages: second }); + assert.equal(JSON.stringify(second), JSON.stringify(first)); + }); + it("R9 a memo save failure still applies the reduction", async () => { + const env = await setup(); + await mkdir(join(env.stateDir, "tokenpilot"), { recursive: true }); + await writeFile(join(env.stateDir, "tokenpilot", "reduction-memo"), "not a directory"); + const code = Array.from({ length: 400 }, (_, i) => `export function handler${i}(input: string): string {\n return input.trim() + "${i}";\n}\n`).join("\n"); + // A trimmed file read records a disclosure, so the memo has something to save. + const readHistory = (): OcMessageWithParts[] => [ + { info: { id: "msg_u1", sessionID: S, role: "user", model: { providerID: "llama", modelID: "qwen3" } }, parts: [{ id: "p_u1", sessionID: S, messageID: "msg_u1", type: "text", text: "read the handlers" }] }, + { + info: { id: "msg_a1", sessionID: S, role: "assistant" }, + parts: [{ id: "p_r1", sessionID: S, messageID: "msg_a1", type: "tool", callID: "call_r", tool: "read", state: { status: "completed", input: { filePath: "/work/src/handlers.ts" }, output: code, title: "read", metadata: {}, time: { start: 1, end: 2 } } }], + }, + ]; + const messages = readHistory(); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages }); + assert.ok(JSON.stringify(messages).length < JSON.stringify(readHistory()).length); + assert.match(await readFile(adapterLogPath(env.stateDir), "utf8"), /reduction memo save failed/); + }); +}); + +describe("eviction overlay", () => { + const TASK: SemanticTaskUpdate = { + taskId: "task-build", objective: "fix the build", lifecycle: "completed", + coveredTurnAbsIds: [`${S}:t1`], completionEvidence: ["build passed"], + }; + const EVICTION_ON = { + modules: { eviction: true }, + eviction: { enabled: true, minBlockChars: 256 }, + taskStateEstimator: { baseUrl: "http://127.0.0.1:1/v1", apiKey: "k", model: "m" }, + }; + function store() { + let current: SessionTaskRegistry = createEmptySessionTaskRegistry(S); + return { load: () => current, persist: (registry: SessionTaskRegistry) => { current = registry; } }; + } + function twoTurns(): OcMessageWithParts[] { + return [ + ...history(), + { info: { id: "msg_u2", sessionID: S, role: "user", model: { providerID: "llama", modelID: "qwen3" } }, parts: [{ id: "p_u2", sessionID: S, messageID: "msg_u2", type: "text", text: "now the docs" }] }, + ]; + } + const partOutput = (messages: OcMessageWithParts[]) => ((messages[1]!.parts[0] as OcToolPart).state as { output: string }).output; + + it("E1 the default config never creates an estimator", async () => { + let created = false; + const env = await setup({}, { createEstimator: () => { created = true; return undefined; } }); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages: twoTurns() }); + assert.equal(created, false); + assert.equal(existsSync(overlayPath(env.stateDir, S)), false); + }); + it("E2 a completed earlier turn is evicted through the overlay", async () => { + const env = await setup(EVICTION_ON, { createEstimator: (): TaskStateEstimator => ({ estimate: () => ({ baseVersion: 0, taskUpdates: [TASK] }) }), registryStore: store() }); + const messages = twoTurns(); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages }); + assert.ok(partOutput(messages).startsWith("[evicted: tool_result part:p_t1]")); + const overlay = JSON.parse(await readFile(overlayPath(env.stateDir, S), "utf8")); + assert.deepEqual(Object.keys(overlay.entries), ["part:p_t1"]); + }); + it("E3 later requests re-apply the stored overlay without estimating again", async () => { + let calls = 0; + const registry = store(); + const env = await setup(EVICTION_ON, { createEstimator: () => ({ estimate: () => { calls += 1; return { baseVersion: 0, taskUpdates: [TASK] }; } }), registryStore: registry }); + const first = twoTurns(); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages: first }); + const second = twoTurns(); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages: second }); + assert.equal(calls, 1); + assert.equal(partOutput(second), partOutput(first)); + }); + it("E4 an overlay save failure defers the eviction", async () => { + const env = await setup(EVICTION_ON, { createEstimator: () => ({ estimate: () => ({ baseVersion: 0, taskUpdates: [TASK] }) }), registryStore: store() }); + const blocked = dirname(overlayPath(env.stateDir, S)); + await mkdir(dirname(blocked), { recursive: true }); + await writeFile(blocked, "not a directory"); + const messages = twoTurns(); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages }); + assert.ok(!partOutput(messages).startsWith("[evicted:")); + await chmod(blocked, 0o644); + }); + it("E5 a throwing estimator skips eviction but reduction still runs", async () => { + const env = await setup(EVICTION_ON, { createEstimator: () => ({ estimate: () => { throw new Error("estimator down"); } }), registryStore: store() }); + const messages = twoTurns(); + await env.hooks["experimental.chat.messages.transform"]!({}, { messages }); + const output = partOutput(messages); + assert.ok(!output.startsWith("[evicted:")); + assert.ok(output.length < BIG.length, "reduction should still apply"); + assert.match(await readFile(adapterLogPath(env.stateDir), "utf8"), /eviction failed open Error: estimator down/); + }); +}); diff --git a/components/adapters/opencode/tests/shared-canonical-before-call.test.ts b/components/adapters/opencode/tests/shared-canonical-before-call.test.ts new file mode 100644 index 00000000..1e4153a9 --- /dev/null +++ b/components/adapters/opencode/tests/shared-canonical-before-call.test.ts @@ -0,0 +1,218 @@ +/** + * Regression matrix: adapters/shared/canonical/before-call.ts + * + * applyCanonicalStablePrefix(instructions, config) + * P1 modules.stabilizer off → unchanged + * P2 blank instructions → unchanged + * P3 no volatile lines → unchanged + * P4 target developer + volatile lines → volatile lines moved to the tail; + * equals shared applyStablePrefixToInstructions + * P5 target user + volatile lines → removed from instructions, returned as userPrefix; + * equals shared applyStablePrefixToInstructions + * P6 dynamicContextText equals the shared rewriteTextForStablePrefix output + * recordCanonicalStability + * V1 unchanged result → writes nothing + * V2 changed result → appends a stability visual snapshot for the session + * runCanonicalBeforeCallReduction (returns messages only; system text is owned by host hooks) + * C1 savings > 0 → returns reduced messages and records a ux-effect for the session + * C2 no savings → returns the input messages array and records nothing + * C3 returned messages equal reduceCanonicalEnvelope's (no extra rewriting) + * canonicalRecoveryProtocolText / withCanonicalRecoveryProtocol + * RP1 text equals what the shared defaultInjectRecoveryProtocol appends + * RP2 appended to a non-empty system prompt after a blank line + * RP3 already present → unchanged (idempotent) + * RP4 empty system prompt → the protocol alone + * RP5 TOKENPILOT_DISABLE_RECOVERY_PROTOCOL set → empty text, prompt unchanged + * CANONICAL_IDENTITY_CODEC / createCanonicalEnvelope + * I1 encode/decode are identities + * I2 envelope fields: host identity, session id, optional turn id, single session mode + */ +import { describe, it, before } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp, readdir } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { configureStatePathResolver } from "@lightrsi/artifact-store"; +import { createStaticStatePathResolver, defaultInjectRecoveryProtocol, readLatestUxEffect } from "@lightrsi/host-adapter"; +import { applyStablePrefixToInstructions, rewriteTextForStablePrefix } from "@lightrsi/stabilizer"; + +import { + CANONICAL_IDENTITY_CODEC, + applyCanonicalStablePrefix, + canonicalRecoveryProtocolText, + withCanonicalRecoveryProtocol, + createCanonicalEnvelope, + recordCanonicalStability, + runCanonicalBeforeCallReduction, +} from "../../shared/canonical/before-call.js"; +import { normalizeCanonicalAdapterConfig } from "../../shared/canonical/config.js"; +import { reduceCanonicalEnvelope } from "../../shared/canonical/reduction.js"; + +const VOLATILE_PROMPT = [ + "You are a coding agent.", + "Today's date: Mon Sep 28 2026", + "Follow the repository conventions.", +].join("\n"); +const STABLE_PROMPT = "You are a coding agent.\nFollow the repository conventions."; +const BIG = Array.from({ length: 3000 }, (_, i) => `line ${i} output from build step ok`).join("\n"); + +async function freshState(): Promise { + const dir = await mkdtemp(join(tmpdir(), "tp-bc-")); + configureStatePathResolver(createStaticStatePathResolver({ hostId: "opencode", displayName: "OpenCode", stateDir: dir, namespaceDir: "tokenpilot" })); + return dir; +} + +async function listFiles(root: string): Promise { + const out: string[] = []; + for (const entry of await readdir(root, { withFileTypes: true, recursive: true })) { + if (entry.isFile()) out.push(join(entry.parentPath ?? "", entry.name)); + } + return out; +} + +const cfg = (stateDir: string, raw: Record = {}) => normalizeCanonicalAdapterConfig({ ...raw, stateDir }, { defaultStateDir: stateDir }); + +describe("applyCanonicalStablePrefix", () => { + let stateDir = ""; + before(async () => { stateDir = await freshState(); }); + + it("P1 stabilizer off leaves the prompt unchanged", () => { + const out = applyCanonicalStablePrefix(VOLATILE_PROMPT, cfg(stateDir, { modules: { stabilizer: false } })); + assert.deepEqual(out, { changed: false, instructions: VOLATILE_PROMPT, dynamicContextText: "" }); + }); + it("P2 blank instructions are unchanged", () => { + assert.equal(applyCanonicalStablePrefix(" ", cfg(stateDir)).changed, false); + }); + it("P3 a prompt without volatile lines is unchanged", () => { + assert.deepEqual(applyCanonicalStablePrefix(STABLE_PROMPT, cfg(stateDir)), { changed: false, instructions: STABLE_PROMPT, dynamicContextText: "" }); + }); + it("P4 developer target moves volatile lines to the tail, like the shared stabilizer", () => { + const out = applyCanonicalStablePrefix(VOLATILE_PROMPT, cfg(stateDir)); + const shared = applyStablePrefixToInstructions({ + envelope: { session: { host: { hostId: "x" } }, model: "", instructions: VOLATILE_PROMPT, messages: [{ role: "user", content: "" }] }, + dynamicContextTarget: "developer", + mergeDynamicContextIntoInstructions: true, + }); + assert.equal(out.changed, true); + assert.equal(out.instructions, shared.instructions); + assert.ok(out.instructions.startsWith(STABLE_PROMPT)); + assert.ok(out.instructions.endsWith("Today's date: Mon Sep 28 2026")); + assert.equal(out.userPrefix, undefined); + }); + it("P5 user target removes volatile lines and returns them for the first user message", () => { + const out = applyCanonicalStablePrefix(VOLATILE_PROMPT, cfg(stateDir, { hooks: { dynamicContextTarget: "user" } })); + const shared = applyStablePrefixToInstructions({ + envelope: { session: { host: { hostId: "x" } }, model: "", instructions: VOLATILE_PROMPT, messages: [{ role: "user", content: "" }] }, + dynamicContextTarget: "user", + }); + assert.equal(out.instructions, shared.instructions); + assert.ok(!out.instructions.includes("Today's date")); + assert.equal(out.userPrefix, "Today's date: Mon Sep 28 2026"); + }); + it("P6 dynamicContextText matches the shared rewrite", () => { + assert.equal(applyCanonicalStablePrefix(VOLATILE_PROMPT, cfg(stateDir)).dynamicContextText, rewriteTextForStablePrefix(VOLATILE_PROMPT).dynamicContextText); + }); +}); + +describe("recordCanonicalStability", () => { + it("V1 an unchanged result writes nothing", async () => { + const stateDir = await freshState(); + await recordCanonicalStability({ + config: cfg(stateDir), sessionId: "s", model: "m", before: STABLE_PROMPT, + result: { changed: false, instructions: STABLE_PROMPT, dynamicContextText: "" }, + }); + assert.deepEqual(await listFiles(stateDir), []); + }); + it("V2 a changed result appends a stability snapshot", async () => { + const stateDir = await freshState(); + const result = applyCanonicalStablePrefix(VOLATILE_PROMPT, cfg(stateDir)); + await recordCanonicalStability({ config: cfg(stateDir), sessionId: "s-stab", model: "m", before: VOLATILE_PROMPT, result }); + const files = await listFiles(stateDir); + assert.ok(files.some((file) => file.includes("stability")), files.join(",")); + }); +}); + +describe("runCanonicalBeforeCallReduction", () => { + const toolEnvelope = (sessionId: string, text: string) => createCanonicalEnvelope({ + hostId: "opencode", displayName: "OpenCode", sessionId, model: "m", + messages: [ + { role: "user", content: "build" }, + { role: "assistant", content: [{ type: "tool_call", toolCallId: "c1", toolName: "bash" }] }, + { role: "tool", content: [{ type: "tool_result", toolCallId: "c1", toolName: "bash", status: "success", text }] }, + ], + }); + + it("C1 savings return reduced messages and record a ux-effect", async () => { + const stateDir = await freshState(); + const env = toolEnvelope("s-ux", BIG); + const out = await runCanonicalBeforeCallReduction({ envelope: env, config: cfg(stateDir) }); + assert.ok((out.summary?.savedChars ?? 0) > 0); + assert.notEqual(out.messages, env.messages); + const latest = await readLatestUxEffect(stateDir); + assert.equal(latest?.sessionId, "s-ux"); + assert.equal(latest?.savedCount, out.summary?.savedChars); + }); + it("C2 no savings return the input messages and record nothing", async () => { + const stateDir = await freshState(); + const env = toolEnvelope("s-none", "short"); + const out = await runCanonicalBeforeCallReduction({ envelope: env, config: cfg(stateDir) }); + assert.equal(out.messages, env.messages); + assert.equal(await readLatestUxEffect(stateDir), null); + }); + it("C3 the result equals reduceCanonicalEnvelope's modulo the archive path", async () => { + const stateDir = await freshState(); + const env = toolEnvelope("s-eq", BIG); + const viaPipeline = await runCanonicalBeforeCallReduction({ envelope: env, config: cfg(stateDir) }); + const direct = await reduceCanonicalEnvelope({ envelope: env, config: cfg(stateDir) }); + const strip = (value: unknown) => JSON.stringify(value).replace(/Archive: [^"\\]*/g, "Archive: "); + assert.equal(strip(viaPipeline.messages), strip(direct.envelope.messages)); + }); +}); + +describe("recovery protocol", () => { + const sharedText = () => defaultInjectRecoveryProtocol(createCanonicalEnvelope({ + hostId: "x", displayName: "x", sessionId: "s", model: "m", instructions: "", messages: [], + })).instructions; + it("RP1 text equals the shared injector's", () => { + assert.equal(canonicalRecoveryProtocolText(), sharedText()); + assert.ok(canonicalRecoveryProtocolText().startsWith("[Recovery Protocol]")); + }); + it("RP2 appended after a blank line to a non-empty prompt", () => { + assert.deepEqual(withCanonicalRecoveryProtocol("SYS"), { changed: true, text: `SYS\n\n${canonicalRecoveryProtocolText()}` }); + }); + it("RP3 idempotent when already present", () => { + const once = withCanonicalRecoveryProtocol("SYS").text; + assert.deepEqual(withCanonicalRecoveryProtocol(once), { changed: false, text: once }); + }); + it("RP4 an empty prompt becomes the protocol alone", () => { + assert.deepEqual(withCanonicalRecoveryProtocol(""), { changed: true, text: canonicalRecoveryProtocolText() }); + }); + it("RP5 the disable env var turns injection off", () => { + const previous = process.env.TOKENPILOT_DISABLE_RECOVERY_PROTOCOL; + process.env.TOKENPILOT_DISABLE_RECOVERY_PROTOCOL = "1"; + try { + assert.equal(canonicalRecoveryProtocolText(), ""); + assert.deepEqual(withCanonicalRecoveryProtocol("SYS"), { changed: false, text: "SYS" }); + } finally { + if (previous === undefined) delete process.env.TOKENPILOT_DISABLE_RECOVERY_PROTOCOL; + else process.env.TOKENPILOT_DISABLE_RECOVERY_PROTOCOL = previous; + } + }); +}); + +describe("identity codec and envelope factory", () => { + it("I1 encode and decode are identities", () => { + const env = createCanonicalEnvelope({ hostId: "opencode", displayName: "OpenCode", sessionId: "s", model: "m", messages: [] }); + assert.equal(CANONICAL_IDENTITY_CODEC.encodeRequest(env), env); + assert.equal(CANONICAL_IDENTITY_CODEC.decodeRequest(env), env); + }); + it("I2 envelope fields", () => { + const withTurn = createCanonicalEnvelope({ hostId: "opencode", displayName: "OpenCode", sessionId: "s", turnId: "t1", model: "m", instructions: "sys", messages: [] }); + assert.deepEqual(withTurn.session, { host: { hostId: "opencode", displayName: "OpenCode" }, sessionId: "s", turnId: "t1", sessionMode: "single" }); + assert.equal(withTurn.instructions, "sys"); + const withoutTurn = createCanonicalEnvelope({ hostId: "opencode", displayName: "OpenCode", sessionId: "s", model: "m", messages: [] }); + assert.equal("turnId" in withoutTurn.session, false); + assert.equal("instructions" in withoutTurn, false); + }); +}); diff --git a/components/adapters/opencode/tests/shared-canonical-config.test.ts b/components/adapters/opencode/tests/shared-canonical-config.test.ts new file mode 100644 index 00000000..92226287 --- /dev/null +++ b/components/adapters/opencode/tests/shared-canonical-config.test.ts @@ -0,0 +1,220 @@ +/** + * Regression matrix: adapters/shared/canonical/config.ts + * + * normalizeCanonicalAdapterConfig + * G1 empty object → exact documented defaults (normal mode, eviction off) + * G2 non-object input (null, array, string, number) → same defaults + * G3 booleans: wrong type → default; right type → used + * G4 numbers: below min → min; above max → max; non-finite → default; + * numeric string accepted; fractional truncated + * G5 strings trimmed; blank → undefined + * G6 stateDir: "~/x" expanded against $HOME; absent → host default + * G7 dynamicContextTarget: "user" → user; anything else → developer + * G8 logLevel: "debug" → debug; anything else → info + * G9 passOptions: only known pass names with non-empty objects are kept + * G10 estimator enums: each valid value kept; invalid → undefined + * load / write + * L1 missing file → defaults (with host default stateDir) + * L2 existing file → normalized contents + * L3 malformed JSON → throws (callers decide how to fail open) + * W1 write → atomic, mode 0600, round-trips through load + * evaluateEvictionReadiness (checks in this order) + * R1 host binding lacks eviction → unsupported_on_host + * R2 modules.eviction off → module_disabled + * R3 eviction.enabled off → eviction_disabled + * R4 taskStateEstimator.enabled === false → estimator_disabled + * R5 any of baseUrl/model/apiKey missing → estimator_incomplete (+ missing list) + * R6 everything set → ready, active + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp, readFile, stat, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { + evaluateEvictionReadiness, + loadCanonicalAdapterConfig, + normalizeCanonicalAdapterConfig, + writeCanonicalAdapterConfig, + type CanonicalAdapterConfig, +} from "../../shared/canonical/config.js"; + +const DEFAULT_STATE = "/state/default"; +const norm = (raw: unknown) => normalizeCanonicalAdapterConfig(raw, { defaultStateDir: DEFAULT_STATE }); + +const DEFAULTS: CanonicalAdapterConfig = { + enabled: true, + logLevel: "info", + stateDir: DEFAULT_STATE, + hooks: { dynamicContextTarget: "developer" }, + modules: { stabilizer: true, reduction: true, eviction: false }, + eviction: { enabled: false, minBlockChars: 4000, failureMode: "bypass" }, + reduction: { + triggerMinChars: 2200, + maxToolChars: 1200, + passes: { + readStateCompaction: true, + toolPayloadTrim: true, + htmlSlimming: true, + execOutputTruncation: true, + agentsStartupOptimization: true, + }, + passOptions: {}, + }, + taskStateEstimator: { + enabled: undefined, + baseUrl: undefined, + apiKey: undefined, + model: undefined, + requestTimeoutMs: 60_000, + batchTurns: 5, + evictionLookaheadTurns: 3, + inputMode: undefined, + lifecycleMode: undefined, + evidenceMode: undefined, + }, +}; + +describe("normalizeCanonicalAdapterConfig", () => { + it("G1 an empty object yields the documented defaults", () => { + assert.deepEqual(norm({}), DEFAULTS); + }); + it("G2 non-object input yields the same defaults", () => { + for (const raw of [null, undefined, [], "x", 3]) assert.deepEqual(norm(raw), DEFAULTS, String(raw)); + }); + it("G3 booleans of the wrong type fall back; booleans are used", () => { + assert.equal(norm({ enabled: "false" }).enabled, true); + assert.equal(norm({ enabled: false }).enabled, false); + assert.equal(norm({ modules: { eviction: 1 } }).modules.eviction, false); + assert.equal(norm({ modules: { eviction: true } }).modules.eviction, true); + }); + it("G4 numbers are clamped, parsed and truncated", () => { + assert.equal(norm({ reduction: { triggerMinChars: 1 } }).reduction.triggerMinChars, 256); + assert.equal(norm({ reduction: { triggerMinChars: 9e9 } }).reduction.triggerMinChars, 1_000_000); + assert.equal(norm({ reduction: { triggerMinChars: "abc" } }).reduction.triggerMinChars, 2200); + assert.equal(norm({ reduction: { triggerMinChars: "3000" } }).reduction.triggerMinChars, 3000); + assert.equal(norm({ reduction: { maxToolChars: 1500.9 } }).reduction.maxToolChars, 1500); + assert.equal(norm({ eviction: { minBlockChars: 10 } }).eviction.minBlockChars, 256); + assert.equal(norm({ taskStateEstimator: { requestTimeoutMs: 1 } }).taskStateEstimator.requestTimeoutMs, 1000); + }); + it("G5 strings are trimmed and blanks become undefined", () => { + const cfg = norm({ taskStateEstimator: { baseUrl: " http://x ", apiKey: " ", model: "m" } }); + assert.equal(cfg.taskStateEstimator.baseUrl, "http://x"); + assert.equal(cfg.taskStateEstimator.apiKey, undefined); + }); + it("G6 stateDir expands ~ and defaults to the host directory", () => { + const previous = process.env.HOME; + process.env.HOME = "/home/tester"; + try { + assert.equal(norm({ stateDir: "~/tp" }).stateDir, "/home/tester/tp"); + assert.equal(norm({ stateDir: "~" }).stateDir, "/home/tester"); + } finally { + process.env.HOME = previous; + } + assert.equal(norm({ stateDir: " " }).stateDir, DEFAULT_STATE); + }); + it("G7 dynamicContextTarget accepts user and defaults to developer", () => { + assert.equal(norm({ hooks: { dynamicContextTarget: "user" } }).hooks.dynamicContextTarget, "user"); + assert.equal(norm({ hooks: { dynamicContextTarget: "system" } }).hooks.dynamicContextTarget, "developer"); + }); + it("G8 logLevel accepts debug and defaults to info", () => { + assert.equal(norm({ logLevel: "debug" }).logLevel, "debug"); + assert.equal(norm({ logLevel: "trace" }).logLevel, "info"); + }); + it("G9 passOptions keep only known passes with non-empty objects", () => { + const cfg = norm({ + reduction: { + passOptions: { + toolPayloadTrim: { maxChars: 10 }, + htmlSlimming: {}, + bogusPass: { a: 1 }, + execOutputTruncation: "x", + }, + }, + }); + assert.deepEqual(cfg.reduction.passOptions, { toolPayloadTrim: { maxChars: 10 } }); + }); + it("G10 estimator enums keep valid values and drop invalid ones", () => { + for (const inputMode of ["sliding_window", "completed_summary_plus_active_turns"] as const) { + assert.equal(norm({ taskStateEstimator: { inputMode } }).taskStateEstimator.inputMode, inputMode); + } + for (const lifecycleMode of ["coupled", "decoupled"] as const) { + assert.equal(norm({ taskStateEstimator: { lifecycleMode } }).taskStateEstimator.lifecycleMode, lifecycleMode); + } + for (const evidenceMode of ["two_state", "three_state"] as const) { + assert.equal(norm({ taskStateEstimator: { evidenceMode } }).taskStateEstimator.evidenceMode, evidenceMode); + } + const bad = norm({ taskStateEstimator: { inputMode: "x", lifecycleMode: "x", evidenceMode: "x" } }).taskStateEstimator; + assert.deepEqual([bad.inputMode, bad.lifecycleMode, bad.evidenceMode], [undefined, undefined, undefined]); + }); +}); + +describe("load and write", () => { + it("L1 a missing file yields defaults with the host state dir", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-cfg-")); + assert.deepEqual(await loadCanonicalAdapterConfig(join(dir, "none.json"), { defaultStateDir: DEFAULT_STATE }), DEFAULTS); + }); + it("L2 an existing file is normalized", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-cfg-")); + const path = join(dir, "tokenpilot.json"); + await writeFile(path, JSON.stringify({ reduction: { maxToolChars: 1800 } })); + const cfg = await loadCanonicalAdapterConfig(path, { defaultStateDir: DEFAULT_STATE }); + assert.equal(cfg.reduction.maxToolChars, 1800); + assert.equal(cfg.reduction.triggerMinChars, 2200); + }); + it("L3 malformed JSON throws", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-cfg-")); + const path = join(dir, "tokenpilot.json"); + await writeFile(path, "{not json"); + await assert.rejects(loadCanonicalAdapterConfig(path, { defaultStateDir: DEFAULT_STATE })); + }); + it("W1 write is atomic, private, and round-trips", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-cfg-")); + const path = join(dir, "nested", "tokenpilot.json"); + const cfg = norm({ reduction: { maxToolChars: 1800 } }); + await writeCanonicalAdapterConfig(cfg, path); + assert.deepEqual(await loadCanonicalAdapterConfig(path, { defaultStateDir: DEFAULT_STATE }), cfg); + assert.equal((await stat(path)).mode & 0o777, 0o600); + assert.ok((await readFile(path, "utf8")).endsWith("}\n")); + }); +}); + +describe("evaluateEvictionReadiness", () => { + const ready = () => norm({ + modules: { eviction: true }, + eviction: { enabled: true }, + taskStateEstimator: { baseUrl: "http://x", model: "m", apiKey: "k" }, + }); + it("R1 an unsupported host is never active, even when fully configured", () => { + assert.deepEqual(evaluateEvictionReadiness(ready(), false), { active: false, reason: "unsupported_on_host", missing: [] }); + }); + it("R2 modules.eviction off", () => { + const cfg = ready(); + cfg.modules.eviction = false; + assert.equal(evaluateEvictionReadiness(cfg, true).reason, "module_disabled"); + }); + it("R3 eviction.enabled off", () => { + const cfg = ready(); + cfg.eviction.enabled = false; + assert.equal(evaluateEvictionReadiness(cfg, true).reason, "eviction_disabled"); + }); + it("R4 estimator explicitly disabled", () => { + const cfg = ready(); + cfg.taskStateEstimator.enabled = false; + assert.equal(evaluateEvictionReadiness(cfg, true).reason, "estimator_disabled"); + }); + it("R5 each missing estimator field is reported", () => { + for (const key of ["baseUrl", "model", "apiKey"] as const) { + const cfg = ready(); + cfg.taskStateEstimator[key] = undefined; + assert.deepEqual(evaluateEvictionReadiness(cfg, true), { active: false, reason: "estimator_incomplete", missing: [key] }); + } + }); + it("R6 a complete configuration on a supporting host is active", () => { + assert.deepEqual(evaluateEvictionReadiness(ready(), true), { active: true, reason: "ready", missing: [] }); + }); + it("defaults are inactive (eviction is opt-in)", () => { + assert.equal(evaluateEvictionReadiness(DEFAULTS, true).active, false); + }); +}); diff --git a/components/adapters/opencode/tests/shared-canonical-eviction.test.ts b/components/adapters/opencode/tests/shared-canonical-eviction.test.ts new file mode 100644 index 00000000..62bbbc16 --- /dev/null +++ b/components/adapters/opencode/tests/shared-canonical-eviction.test.ts @@ -0,0 +1,281 @@ +/** + * Regression matrix: adapters/shared/canonical/eviction.ts + * + * surfaceItemKind — one case per class + * K1 checkpoint → compaction_checkpoint K2 tool role → tool_result + * K3 assistant with tool_call → tool_call K4 assistant text only → message + * K5 user → message + * buildCanonicalRawSemanticSnapshot + * N1 user/assistant text → message records anchored to their turn + * N2 blank text → no message record + * N3 tool_call → tool call record (argumentsText from JSON args, summary ≤ 600 chars) + * N4 tool_result → result record (status, tool name from block, else from call) + * N5 lastTurnSeq = max turn + * buildCanonicalSurfaceItems + * I1 ids, turns, kinds and chars carried through + * I2 callIds only on tool_call / tool_result items + * buildEvictionReplacement + * X1 stub text `[evicted: ]` + shared recovery hint, dataKey `evicted:` + * X2 the original is recoverable via the shared resolveMemoryFaultRecover + * X3 empty original → stub only, no archive + * X4 archive write failure → throws (never evict without a recoverable original) + * createCanonicalEstimator + * E1 incomplete estimator config → undefined; E2 complete → estimator + * runCanonicalSurfaceEviction (fake estimator, in-memory registry store) + * V1 completed earlier turn → its large tool result is replaced; call envelope kept + * V2 alreadyEvicted entries are neither re-evicted nor counted as effective + * V3 checkpoints are never evicted + * V4 eviction.enabled=false → apply never called, registry still updated + * V5 host apply reports deferred → no replacements returned, watermark behind + * V6 archive failure → batch deferred, apply never called + * V7 current turn is never evicted + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { configureStatePathResolver } from "@lightrsi/artifact-store"; +import type { SemanticTaskUpdate } from "@lightrsi/eviction"; +import { createEmptySessionTaskRegistry, type SessionTaskRegistry } from "@lightrsi/history"; +import { createStaticStatePathResolver } from "@lightrsi/host-adapter"; +import type { RuntimeMessage } from "@lightrsi/kernel"; +import { resolveMemoryFaultRecover } from "@lightrsi/mcp"; + +import { normalizeCanonicalAdapterConfig } from "../../shared/canonical/config.js"; +import { + buildCanonicalRawSemanticSnapshot, + buildCanonicalSurfaceItems, + buildEvictionReplacement, + createCanonicalEstimator, + runCanonicalSurfaceEviction, + surfaceItemKind, + type CanonicalSurfaceEntry, +} from "../../shared/canonical/eviction.js"; + +const SESSION = "s-ev"; +const BIG = "x".repeat(6000); + +async function freshState(): Promise { + const dir = await mkdtemp(join(tmpdir(), "tp-ev-")); + configureStatePathResolver(createStaticStatePathResolver({ hostId: "opencode", displayName: "OpenCode", stateDir: dir, namespaceDir: "tokenpilot" })); + return dir; +} + +const user = (text: string): RuntimeMessage => ({ role: "user", content: text }); +const assistantText = (text: string): RuntimeMessage => ({ role: "assistant", content: [{ type: "text", text }] }); +const call = (id: string, toolName = "read"): RuntimeMessage => ({ + role: "assistant", + content: [{ type: "tool_call", toolCallId: id, toolName, argumentsJson: { path: "/cfg" } }], +}); +const result = (id: string, text: string, toolName?: string): RuntimeMessage => ({ + role: "tool", + content: [{ type: "tool_result", toolCallId: id, ...(toolName ? { toolName } : {}), status: "success", text }], +}); + +/** Turn 1 finished a task with a big tool result; turn 2 is current. */ +function surface(): CanonicalSurfaceEntry[] { + return [ + { id: "u1", turn: 1, message: user("read the config") }, + { id: "a1", turn: 1, message: call("c1") }, + { id: "r1", turn: 1, message: result("c1", BIG) }, + { id: "a1b", turn: 1, message: assistantText(`config says x. ${"detail ".repeat(700)}`) }, + { id: "u2", turn: 2, message: user("now something else") }, + { id: "r2", turn: 2, message: result("c2", BIG) }, + ]; +} + +function completedEstimator(): { estimate(): { baseVersion: number; taskUpdates: SemanticTaskUpdate[] } } { + return { + estimate: () => ({ + baseVersion: 0, + taskUpdates: [{ + taskId: "task-config", + objective: "read the config", + lifecycle: "completed", + coveredTurnAbsIds: [`${SESSION}:t1`], + completionEvidence: ["reported config value"], + }], + }), + }; +} + +function memoryStore(initial = createEmptySessionTaskRegistry(SESSION)) { + let current: SessionTaskRegistry = initial; + return { + get: () => current, + load: () => current, + persist: (registry: SessionTaskRegistry) => { current = registry; }, + }; +} + +const enabledConfig = (stateDir: string, patch: Record = {}) => normalizeCanonicalAdapterConfig({ + stateDir, + modules: { eviction: true }, + eviction: { enabled: true, minBlockChars: 256 }, + taskStateEstimator: { baseUrl: "http://x", apiKey: "k", model: "m" }, + ...patch, +}, { defaultStateDir: stateDir }); + +describe("surfaceItemKind", () => { + it("K1 checkpoint", () => assert.equal(surfaceItemKind({ id: "x", turn: 1, message: user("s"), checkpoint: true }), "compaction_checkpoint")); + it("K2 tool role", () => assert.equal(surfaceItemKind({ id: "x", turn: 1, message: result("c", "o") }), "tool_result")); + it("K3 assistant with a call", () => assert.equal(surfaceItemKind({ id: "x", turn: 1, message: call("c") }), "tool_call")); + it("K4 assistant text only", () => assert.equal(surfaceItemKind({ id: "x", turn: 1, message: assistantText("t") }), "message")); + it("K5 user", () => assert.equal(surfaceItemKind({ id: "x", turn: 1, message: user("t") }), "message")); +}); + +describe("buildCanonicalRawSemanticSnapshot", () => { + const snap = () => buildCanonicalRawSemanticSnapshot(SESSION, [ + { id: "u1", turn: 1, message: user("read it") }, + { id: "blank", turn: 1, message: assistantText(" ") }, + { id: "a1", turn: 1, message: { role: "assistant", content: [{ type: "tool_call", toolCallId: "c1", toolName: "read", argumentsJson: { path: "p".repeat(900) } }] } }, + { id: "r1", turn: 1, message: result("c1", "out") }, + { id: "r1e", turn: 3, message: { role: "tool", content: [{ type: "tool_result", toolCallId: "c9", toolName: "bash", status: "error", text: "boom" }] } }, + ]); + it("N1 text messages are anchored to their turn", () => { + assert.deepEqual(snap().messages, [{ anchor: { sessionId: SESSION, turnAbsId: `${SESSION}:t1`, turnSeq: 1, role: "user" }, role: "user", text: "read it" }]); + }); + it("N2 blank text produces no record", () => { + assert.equal(snap().messages.length, 1); + }); + it("N3 tool calls carry JSON argument text and a bounded summary", () => { + const record = snap().toolCalls[0]; + assert.equal(record?.toolName, "read"); + assert.ok(record?.argumentsText?.startsWith('{"path":"ppp')); + assert.ok((record?.argumentsSummary.length ?? 0) <= 603); + }); + it("N4 tool results carry status and resolve the tool name from the call", () => { + const [ok, err] = snap().toolResults; + assert.deepEqual([ok?.toolName, ok?.status, ok?.fullText], ["read", "success", "out"]); + assert.deepEqual([err?.toolName, err?.status], ["bash", "error"]); + }); + it("N5 lastTurnSeq is the highest turn", () => { + assert.equal(snap().lastTurnSeq, 3); + }); +}); + +describe("buildCanonicalSurfaceItems", () => { + it("I1/I2 ids, turns, kinds, chars and call ids", () => { + assert.deepEqual(buildCanonicalSurfaceItems(surface().slice(0, 3)), [ + { sourceEventSeq: "u1", turn: 1, kind: "message", chars: 15 }, + { sourceEventSeq: "a1", turn: 1, kind: "tool_call", callIds: ["c1"], chars: 0 }, + { sourceEventSeq: "r1", turn: 1, kind: "tool_result", callIds: ["c1"], chars: 6000 }, + ]); + }); +}); + +describe("buildEvictionReplacement", () => { + it("X1/X2 stub + recovery hint, and the original is recoverable", async () => { + const stateDir = await freshState(); + const replacement = await buildEvictionReplacement({ sessionId: SESSION, entry: { id: "r1", turn: 1, message: result("c1", BIG, "read") } }); + assert.ok(replacement.text.startsWith("[evicted: tool_result r1]")); + assert.equal(replacement.dataKey, "evicted:r1"); + assert.ok(replacement.text.includes('"dataKey":"evicted:r1"')); + assert.equal(replacement.originalChars, BIG.length); + const recovered = await resolveMemoryFaultRecover({ dataKey: "evicted:r1", stateDir }); + assert.ok(recovered.text.includes(BIG.slice(0, 100))); + }); + it("X3 an empty original yields a stub without a hint", async () => { + await freshState(); + const replacement = await buildEvictionReplacement({ sessionId: SESSION, entry: { id: "e", turn: 1, message: result("c", "") } }); + assert.equal(replacement.text, "[evicted: tool_result e]"); + }); + it("X4 an unwritable archive throws", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-ev-")); + const blocker = join(dir, "blocker"); + await writeFile(blocker, "file, not a dir"); + configureStatePathResolver(createStaticStatePathResolver({ hostId: "opencode", displayName: "OpenCode", stateDir: blocker, namespaceDir: "tokenpilot" })); + await assert.rejects(buildEvictionReplacement({ sessionId: SESSION, entry: { id: "r1", turn: 1, message: result("c1", BIG) } })); + }); +}); + +describe("createCanonicalEstimator", () => { + it("E1 incomplete config yields no estimator", () => { + assert.equal(createCanonicalEstimator(normalizeCanonicalAdapterConfig({ taskStateEstimator: { baseUrl: "http://x" } }, { defaultStateDir: "/s" })), undefined); + }); + it("E2 complete config yields an estimator", () => { + assert.equal(typeof createCanonicalEstimator(enabledConfig("/s"))?.estimate, "function"); + }); +}); + +describe("runCanonicalSurfaceEviction", () => { + it("V1 a completed turn's large result and message are replaced; the call envelope is kept", async () => { + const stateDir = await freshState(); + const store = memoryStore(); + let applied: string[] = []; + const cycle = await runCanonicalSurfaceEviction({ + sessionId: SESSION, entries: surface(), config: enabledConfig(stateDir), estimator: completedEstimator(), registryStore: store, + apply(replacements) { applied = replacements.map((r) => r.id); return "committed"; }, + }); + assert.deepEqual(applied, ["r1", "a1b"]); + assert.equal(cycle.status, "applied"); + assert.equal(cycle.result.replacements.length, 2); + assert.equal(store.get().lastProcessedTurnSeq, 2); + }); + it("V2 already-evicted entries are not evicted again", async () => { + const stateDir = await freshState(); + const entries = surface().map((entry) => (entry.id === "r1" ? { ...entry, alreadyEvicted: true } : entry)); + let applied: string[] = []; + await runCanonicalSurfaceEviction({ + sessionId: SESSION, entries, config: enabledConfig(stateDir), estimator: completedEstimator(), registryStore: memoryStore(), + apply(replacements) { applied = replacements.map((r) => r.id); return "committed"; }, + }); + assert.deepEqual(applied, ["a1b"]); + }); + it("V3 checkpoints are never evicted", async () => { + const stateDir = await freshState(); + const entries = surface().map((entry) => (entry.id === "a1b" ? { ...entry, checkpoint: true } : entry)); + let applied: string[] = []; + await runCanonicalSurfaceEviction({ + sessionId: SESSION, entries, config: enabledConfig(stateDir), estimator: completedEstimator(), registryStore: memoryStore(), + apply(replacements) { applied = replacements.map((r) => r.id); return "committed"; }, + }); + assert.deepEqual(applied, ["r1"]); + }); + it("V4 eviction disabled tracks task state without applying", async () => { + const stateDir = await freshState(); + const store = memoryStore(); + let called = false; + await runCanonicalSurfaceEviction({ + sessionId: SESSION, entries: surface(), config: enabledConfig(stateDir, { eviction: { enabled: false } }), estimator: completedEstimator(), registryStore: store, + apply() { called = true; return "committed"; }, + }); + assert.equal(called, false); + assert.deepEqual(store.get().completedTaskIds, ["task-config"]); + }); + it("V5 a deferred host apply returns no replacements and keeps the watermark", async () => { + const stateDir = await freshState(); + const store = memoryStore(); + const cycle = await runCanonicalSurfaceEviction({ + sessionId: SESSION, entries: surface(), config: enabledConfig(stateDir), estimator: completedEstimator(), registryStore: store, + apply: () => "deferred", + }); + assert.equal(cycle.status, "deferred"); + assert.deepEqual(cycle.result.replacements, []); + assert.equal(store.get().lastProcessedTurnSeq, 0); + }); + it("V6 an archive failure defers the batch without calling apply", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-ev-")); + const blocker = join(dir, "blocker"); + await writeFile(blocker, "file"); + configureStatePathResolver(createStaticStatePathResolver({ hostId: "opencode", displayName: "OpenCode", stateDir: blocker, namespaceDir: "tokenpilot" })); + let called = false; + const cycle = await runCanonicalSurfaceEviction({ + sessionId: SESSION, entries: surface(), config: enabledConfig(dir), estimator: completedEstimator(), registryStore: memoryStore(), + apply() { called = true; return "committed"; }, + }); + assert.equal(called, false); + assert.equal(cycle.status, "deferred"); + }); + it("V7 the current turn is never evicted", async () => { + const stateDir = await freshState(); + let applied: string[] = []; + await runCanonicalSurfaceEviction({ + sessionId: SESSION, entries: surface(), config: enabledConfig(stateDir), estimator: completedEstimator(), registryStore: memoryStore(), + apply(replacements) { applied = replacements.map((r) => r.id); return "committed"; }, + }); + assert.ok(!applied.includes("u2") && !applied.includes("r2")); + }); +}); diff --git a/components/adapters/opencode/tests/shared-canonical-logger.test.ts b/components/adapters/opencode/tests/shared-canonical-logger.test.ts new file mode 100644 index 00000000..7b762a90 --- /dev/null +++ b/components/adapters/opencode/tests/shared-canonical-logger.test.ts @@ -0,0 +1,99 @@ +/** + * Regression matrix: adapters/shared/canonical/logger.ts + * + * failOpen(logger, hook, body, fallback) + * F1 body returns synchronously → value, nothing logged + * F2 body resolves → value, nothing logged + * F3 body throws synchronously → fallback, one warn naming the hook + * F4 body rejects → fallback, one warn naming the hook + * createFileLogger + * L1 info/warn append a line with timestamp, level, host id and detail + * L2 debug writes only when debug() is true + * L3 no stateDir → no write, no throw + * L4 unwritable stateDir → no throw + * L5 Error details render as "Name: message" + * adapterLogPath + * P1 /tokenpilot/adapter.log + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { adapterLogPath, createFileLogger, failOpen, type AdapterLogger } from "../../shared/canonical/logger.js"; + +function recordingLogger(): AdapterLogger & { warns: Array<[string, unknown]> } { + const warns: Array<[string, unknown]> = []; + return { warns, info() {}, debug() {}, warn(message, detail) { warns.push([message, detail]); } }; +} + +describe("failOpen", () => { + it("F1 a synchronous value passes through", async () => { + const log = recordingLogger(); + assert.equal(await failOpen(log, "h", () => 1, 0), 1); + assert.equal(log.warns.length, 0); + }); + it("F2 a resolved value passes through", async () => { + const log = recordingLogger(); + assert.equal(await failOpen(log, "h", async () => 2, 0), 2); + assert.equal(log.warns.length, 0); + }); + it("F3 a synchronous throw returns the fallback and warns", async () => { + const log = recordingLogger(); + assert.equal(await failOpen(log, "context", () => { throw new Error("x"); }, "fb"), "fb"); + assert.deepEqual(log.warns.map(([m]) => m), ["context failed open"]); + }); + it("F4 a rejection returns the fallback and warns", async () => { + const log = recordingLogger(); + assert.equal(await failOpen(log, "turn_end", () => Promise.reject(new Error("y")), undefined), undefined); + assert.deepEqual(log.warns.map(([m]) => m), ["turn_end failed open"]); + }); +}); + +describe("createFileLogger", () => { + it("L1 info and warn append formatted lines", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-log-")); + const logger = createFileLogger({ hostId: "opencode", stateDir: () => dir, debug: () => false }); + logger.info("hello", { a: 1 }); + logger.warn("careful"); + const lines = (await readFile(adapterLogPath(dir), "utf8")).trim().split("\n"); + assert.equal(lines.length, 2); + assert.match(lines[0]!, /^\d{4}-\d{2}-\d{2}T\S+ info \[opencode\] hello \{"a":1\}$/); + assert.match(lines[1]!, / warn \[opencode\] careful$/); + }); + it("L2 debug writes only when enabled", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-log-")); + let debug = false; + const logger = createFileLogger({ hostId: "opencode", stateDir: () => dir, debug: () => debug }); + logger.debug("hidden"); + assert.equal(existsSync(adapterLogPath(dir)), false); + debug = true; + logger.debug("shown"); + assert.match(await readFile(adapterLogPath(dir), "utf8"), /debug \[opencode\] shown/); + }); + it("L3 no stateDir writes nothing and does not throw", () => { + const logger = createFileLogger({ hostId: "opencode", stateDir: () => undefined, debug: () => true }); + assert.doesNotThrow(() => logger.warn("x")); + }); + it("L4 an unwritable stateDir does not throw", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-log-")); + const blocker = join(dir, "file"); + await writeFile(blocker, "x"); + const logger = createFileLogger({ hostId: "opencode", stateDir: () => blocker, debug: () => true }); + assert.doesNotThrow(() => logger.warn("x")); + }); + it("L5 errors render as name and message", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-log-")); + const logger = createFileLogger({ hostId: "opencode", stateDir: () => dir, debug: () => false }); + logger.warn("failed", new TypeError("bad input")); + assert.match(await readFile(adapterLogPath(dir), "utf8"), /failed TypeError: bad input/); + }); +}); + +describe("adapterLogPath", () => { + it("P1 lives under the namespace dir", () => { + assert.equal(adapterLogPath("/s"), join("/s", "tokenpilot", "adapter.log")); + }); +}); diff --git a/components/adapters/opencode/tests/shared-canonical-memo-store.test.ts b/components/adapters/opencode/tests/shared-canonical-memo-store.test.ts new file mode 100644 index 00000000..1181cddd --- /dev/null +++ b/components/adapters/opencode/tests/shared-canonical-memo-store.test.ts @@ -0,0 +1,123 @@ +/** + * Regression matrix: adapters/shared/canonical/memo-store.ts + ReductionMemo persistence + * + * ReductionMemo snapshot + * MS1 round trip keeps every disclosed path and its owner (owned and unattributable) + * MS2 fromSnapshot never throws on junk (null, string, wrong or old version, bad + * entries); valid entries survive, invalid ones are dropped; never dirty + * memo-store + * MS3 load: missing file or corrupt JSON → empty memo, no throw + * MS4 save: clean memo → no write; dirty memo → written under + * /tokenpilot/reduction-memo/, dirty cleared, reload equivalent; + * session ids are encoded (no path traversal) + * MS5 end to end: after a host restart the reloaded memo still knows the + * disclosure, so a deliberate re-read (first read compacted away) is sent in + * full; control: a fresh memo without the load trims it + */ +import { describe, it, before } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp, mkdir, readdir, writeFile } from "node:fs/promises"; +import { existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; + +import { configureStatePathResolver } from "@lightrsi/artifact-store"; +import { createStaticStatePathResolver } from "@lightrsi/host-adapter"; +import type { RuntimeMessage } from "@lightrsi/kernel"; + +import { createCanonicalEnvelope } from "../../shared/canonical/before-call.js"; +import { normalizeCanonicalAdapterConfig } from "../../shared/canonical/config.js"; +import { loadReductionMemo, reductionMemoPath, saveReductionMemoIfDirty } from "../../shared/canonical/memo-store.js"; +import { ReductionMemo, reduceCanonicalEnvelope } from "../../shared/canonical/reduction.js"; + +let stateDir = ""; +before(async () => { + stateDir = await mkdtemp(join(tmpdir(), "tp-memo-")); + configureStatePathResolver(createStaticStatePathResolver({ hostId: "opencode", displayName: "OpenCode", stateDir, namespaceDir: "tokenpilot" })); +}); + +const seg = (id: string, path?: string) => ({ id, kind: "volatile", text: "x", priority: 0, metadata: path ? { path } : {} }) as never; + +describe("ReductionMemo snapshot", () => { + it("MS1 a round trip keeps paths and owners", () => { + const memo = new ReductionMemo(); + memo.recordDisclosedReadPaths(["/repo/a.ts", "/repo/b.ts"], [seg("tool-1", "/repo/a.ts")], new Set(["tool-1"])); + const restored = ReductionMemo.fromSnapshot(JSON.parse(JSON.stringify(memo.toSnapshot()))); + assert.deepEqual(restored.disclosedReadPaths, ["/repo/a.ts", "/repo/b.ts"]); + assert.deepEqual(restored.carriedDisclosedReadPaths(new Set(["tool-1"])), ["/repo/b.ts"]); + assert.deepEqual(restored.carriedDisclosedReadPaths(new Set()), ["/repo/a.ts", "/repo/b.ts"]); + assert.equal(restored.dirty, false); + }); + + it("MS2 junk snapshots never throw and keep only valid entries", () => { + for (const junk of [null, undefined, "x", 3, [], { version: 1, disclosed: [["/a", "t1"]] }, { version: 2, disclosed: "nope" }]) { + const memo = ReductionMemo.fromSnapshot(junk); + assert.equal(memo.disclosedReadPaths, undefined); + assert.equal(memo.dirty, false); + } + const mixed = ReductionMemo.fromSnapshot({ + version: 2, + disclosed: [["/Ok", "tool-1"], ["/unowned", null], [7, "x"], "str", ["", "tool-2"]], + }); + assert.deepEqual(mixed.disclosedReadPaths, ["/ok", "/unowned"]); + assert.deepEqual(mixed.carriedDisclosedReadPaths(new Set(["tool-1"])), ["/unowned"]); + assert.equal(mixed.dirty, false); + }); +}); + +describe("memo-store", () => { + it("MS3 a missing or corrupt file loads as an empty memo", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-memo-load-")); + assert.equal((await loadReductionMemo(dir, "none")).disclosedReadPaths, undefined); + await mkdir(dirname(reductionMemoPath(dir, "bad")), { recursive: true }); + await writeFile(reductionMemoPath(dir, "bad"), "{not json"); + const memo = await loadReductionMemo(dir, "bad"); + assert.equal(memo.disclosedReadPaths, undefined); + assert.equal(memo.dirty, false); + }); + + it("MS4 only dirty memos are written, under an encoded session file", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-memo-save-")); + assert.equal(await saveReductionMemoIfDirty(dir, "s", new ReductionMemo()), false); + assert.equal(existsSync(reductionMemoPath(dir, "s")), false); + + const memo = new ReductionMemo(); + memo.recordDisclosedReadPaths(["/repo/a.ts"], [seg("tool-1", "/repo/a.ts")], new Set(["tool-1"])); + const sessionId = "../escape/ses 1"; + assert.equal(await saveReductionMemoIfDirty(dir, sessionId, memo), true); + assert.equal(memo.dirty, false); + assert.equal(dirname(reductionMemoPath(dir, sessionId)), join(dir, "tokenpilot", "reduction-memo")); + assert.deepEqual(await readdir(join(dir, "tokenpilot", "reduction-memo")), [`${encodeURIComponent(sessionId)}.json`]); + assert.deepEqual((await loadReductionMemo(dir, sessionId)).toSnapshot(), memo.toSnapshot()); + assert.equal(await saveReductionMemoIfDirty(dir, sessionId, memo), false, "an unchanged memo is not rewritten"); + }); + + it("MS5 a restarted host still honours a deliberate re-read", async () => { + const CODE = Array.from({ length: 400 }, (_, i) => `export function handler${i}(input: string): string {\n return input.trim() + "${i}";\n}\n`).join("\n"); + const PATH = "/repo/src/handlers.ts"; + const user = (text: string): RuntimeMessage => ({ role: "user", content: [{ type: "text", text }] }); + const read = (callId: string): RuntimeMessage[] => [ + { role: "assistant", content: [{ type: "tool_call", toolCallId: callId, toolName: "read", argumentsJson: { path: PATH } }] }, + { role: "tool", content: [{ type: "tool_result", toolCallId: callId, toolName: "read", status: "success", text: CODE }] }, + ]; + const config = normalizeCanonicalAdapterConfig({}, { defaultStateDir: stateDir }); + const env = (messages: RuntimeMessage[]) => createCanonicalEnvelope({ hostId: "opencode", displayName: "OpenCode", sessionId: "ms5", model: "m", messages }); + const textAt = (envelope: ReturnType, index: number) => { + const block = (envelope.messages[index]?.content as Array<{ text?: string }>)[0]; + return block?.text ?? ""; + }; + + const before = new ReductionMemo(); + const first = await reduceCanonicalEnvelope({ envelope: env([user("read it"), ...read("r1")]), config, memo: before }); + assert.ok(textAt(first.envelope, 2).length < CODE.length, "precondition: the first read is trimmed"); + await saveReductionMemoIfDirty(stateDir, "ms5", before); + + const compacted = [user("summary of earlier work"), user("read it again in full"), ...read("r2")]; + const restarted = await loadReductionMemo(stateDir, "ms5"); + const reread = await reduceCanonicalEnvelope({ envelope: env(compacted), config, memo: restarted }); + assert.equal(textAt(reread.envelope, 3), CODE, "the reloaded memo marks the re-read as a repeat"); + + const control = await reduceCanonicalEnvelope({ envelope: env(compacted), config, memo: new ReductionMemo() }); + assert.ok(textAt(control.envelope, 3).length < CODE.length, "control: without the reload the re-read is trimmed"); + }); +}); diff --git a/components/adapters/opencode/tests/shared-canonical-reduction.test.ts b/components/adapters/opencode/tests/shared-canonical-reduction.test.ts new file mode 100644 index 00000000..a1d4a43a --- /dev/null +++ b/components/adapters/opencode/tests/shared-canonical-reduction.test.ts @@ -0,0 +1,424 @@ +/** + * Regression matrix: adapters/shared/canonical/reduction.ts + * + * ReductionMemo disclosed read paths (carried across requests) + * M1 carriedDisclosedReadPaths: nothing recorded → undefined + * M2 owner segment still present → not carried + * M3 owner segment absent → carried + * M4 unattributable path (no eligible segment has that path) → always carried + * M5 recordDisclosedReadPaths ignores non-arrays, non-strings and blanks; normalizes + * (trim + lowercase); a known path keeps its owner + * M6 only a segment in eligibleOwnerIds (a read trimmed this request) can own a path + * M7 paths missing from the reported set are dropped + * M8 dirty is set only when the recorded set or an owner changes + * M9 clear() forgets everything + * buildCanonicalReductionTurnContext + * B1 only `tool_result` blocks become segments (text/tool_call/string content ignored) + * B2 segment id is `tool-`; positional `message-i-block-j` without a call id + * B3 tool name from the result block, else from the matching tool_call + * B4 path hint from tool_call arguments (path | file_path | filePath | filename) + * B5 pass-through messages are never segments + * B6 empty result text is not a segment but counts as a tool-like message + * B7 latestUserQuery is the last user message's text + * B8 disclosedReadPaths are forwarded into turn metadata + * canonicalEnabledPassIds / canonicalPassOptions + * P1 each pass flag maps to exactly its pass id; all off → empty + * P2 maxToolChars feeds tool_payload_trim; passOptions override per pass + * reduceCanonicalEnvelope (partitioned by skip reason / effect) + * R1 modules.reduction off → unchanged, "disabled" + * R2 no tool results → unchanged, "no_candidate_segments" + * R3 candidates < triggerMinChars → unchanged, "below_trigger_min_chars" + * R4 passes disabled → unchanged, "pipeline_no_effect" + * R5 reduction applied → only tool_result text changes; input not mutated; + * summary counts match the text delta + * R6 equivalence: output text == shared runReductionBeforeCall on the same inputs + * R7 memo on → a second run is byte-identical (cache-stable) + * R8 memo off → a second run is byte-identical too (archive paths are content-derived) + * R9 trimmed content is recoverable through the shared resolveMemoryFaultRecover + * R10 non-text blocks (images, tool_call) and message metadata are preserved + * reduceCanonicalEnvelope across requests with a memo (progressive disclosure) + * D1 a trimmed read still in history stays trimmed and byte-identical on the next + * request (regression: a live smoke run sent it untrimmed → prefix-cache miss) + * D2 once the disclosing read has left history, a fresh read of the same path is a + * repeat read and is left untrimmed (shared semantics preserved) + * D3 two reads of one path in the same history: first trimmed, second untrimmed, + * identical across repeated requests + * D4 when two reads share a path, ownership follows the read actually trimmed; + * dropping an earlier untrimmed read does not make the retained read expand + * D5 a write to the same path earlier in history cannot own the disclosure + * passSavedChars(summary) + * PS1 undefined summary → {} + * PS2 only passes that changed text with savedChars > 0 are listed + * PS3 repeated pass ids are summed + */ +import { describe, it, before } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { configureStatePathResolver } from "@lightrsi/artifact-store"; +import { createStaticStatePathResolver, type HostRequestEnvelope } from "@lightrsi/host-adapter"; +import type { RuntimeMessage } from "@lightrsi/kernel"; +import { resolveMemoryFaultRecover } from "@lightrsi/mcp"; +import { runReductionBeforeCall } from "@lightrsi/reduction"; + +import { createCanonicalEnvelope } from "../../shared/canonical/before-call.js"; +import { normalizeCanonicalAdapterConfig, type CanonicalAdapterConfig } from "../../shared/canonical/config.js"; +import { + ReductionMemo, + buildCanonicalReductionTurnContext, + passSavedChars, + canonicalEnabledPassIds, + canonicalPassOptions, + prepareCanonicalReductionInputs, + reduceCanonicalEnvelope, +} from "../../shared/canonical/reduction.js"; + +let stateDir = ""; +before(async () => { + stateDir = await mkdtemp(join(tmpdir(), "tp-red-")); + configureStatePathResolver(createStaticStatePathResolver({ hostId: "opencode", displayName: "OpenCode", stateDir, namespaceDir: "tokenpilot" })); +}); + +const config = (raw: unknown = {}): CanonicalAdapterConfig => normalizeCanonicalAdapterConfig(raw, { defaultStateDir: stateDir }); +const BIG = Array.from({ length: 3000 }, (_, i) => `line ${i} output from build step ok`).join("\n"); + +function envelope(messages: RuntimeMessage[], sessionId = "s1"): HostRequestEnvelope { + return createCanonicalEnvelope({ hostId: "opencode", displayName: "OpenCode", sessionId, model: "m", messages }); +} + +function toolTurn(callId: string, text: string, toolName = "bash", args: Record = { command: "make" }): RuntimeMessage[] { + return [ + { role: "assistant", content: [{ type: "tool_call", toolCallId: callId, toolName, argumentsJson: args }] }, + { role: "tool", content: [{ type: "tool_result", toolCallId: callId, toolName, status: "success", text }] }, + ]; +} + +function resultText(env: HostRequestEnvelope, index: number): string { + const content = env.messages[index]?.content; + assert.ok(Array.isArray(content)); + const block = content[0]; + assert.ok(block && block.type === "tool_result"); + return block.text; +} + +describe("buildCanonicalReductionTurnContext", () => { + it("B1 only tool_result blocks become segments", () => { + const built = buildCanonicalReductionTurnContext(envelope([ + { role: "user", content: "hi" }, + { role: "assistant", content: [{ type: "text", text: "sure" }] }, + ...toolTurn("c1", "out"), + { role: "tool", content: "raw string content" }, + ])); + assert.deepEqual(built.turnCtx.segments.map((s) => s.id), ["tool-c1"]); + }); + it("B2 segment ids use the call id, else the position", () => { + const built = buildCanonicalReductionTurnContext(envelope([ + { role: "tool", content: [{ type: "tool_result", status: "success", text: "anon" }] }, + ...toolTurn("c9", "named"), + ])); + assert.deepEqual(built.turnCtx.segments.map((s) => s.id), ["message-0-block-0", "tool-c9"]); + }); + it("B3 tool name comes from the result block, else the matching call", () => { + const built = buildCanonicalReductionTurnContext(envelope([ + { role: "assistant", content: [{ type: "tool_call", toolCallId: "c1", toolName: "grep" }] }, + { role: "tool", content: [{ type: "tool_result", toolCallId: "c1", status: "success", text: "x" }] }, + { role: "tool", content: [{ type: "tool_result", toolCallId: "c1", toolName: "own", status: "success", text: "y" }] }, + ])); + assert.deepEqual(built.bindings.map((b) => b.toolName), ["grep", "own"]); + }); + it("B4 path hints come from path, file_path, filePath or filename", () => { + for (const key of ["path", "file_path", "filePath", "filename"]) { + const built = buildCanonicalReductionTurnContext(envelope(toolTurn("c1", "x", "read", { [key]: " /src/a.ts " }))); + assert.equal(built.turnCtx.segments[0]?.metadata?.path, "/src/a.ts", key); + } + }); + it("B5 pass-through messages are never segments", () => { + const built = buildCanonicalReductionTurnContext(envelope([ + { role: "tool", content: [{ type: "tool_result", toolCallId: "c1", status: "success", text: "x" }], metadata: { lightrsiPassThrough: true } }, + ])); + assert.equal(built.turnCtx.segments.length, 0); + }); + it("B6 empty result text is skipped but counted as tool-like", () => { + const built = buildCanonicalReductionTurnContext(envelope(toolTurn("c1", ""))); + assert.equal(built.turnCtx.segments.length, 0); + assert.equal(built.diagnostics.toolLikeMessages, 1); + }); + it("B7 latestUserQuery is the last user text", () => { + const built = buildCanonicalReductionTurnContext(envelope([ + { role: "user", content: "first" }, + { role: "user", content: [{ type: "text", text: "second" }] }, + ...toolTurn("c1", "x"), + ])); + assert.equal(built.turnCtx.metadata?.latestUserQuery, "second"); + }); + it("B8 disclosed read paths are forwarded", () => { + const built = buildCanonicalReductionTurnContext(envelope(toolTurn("c1", "x")), { disclosedReadPaths: ["/a"] }); + assert.deepEqual(built.turnCtx.metadata?.disclosedReadPaths, ["/a"]); + }); +}); + +describe("pass selection", () => { + const names = { + readStateCompaction: "read_state_compaction", + toolPayloadTrim: "tool_payload_trim", + htmlSlimming: "html_slimming", + execOutputTruncation: "exec_output_truncation", + agentsStartupOptimization: "agents_startup_optimization", + } as const; + const allOff = Object.fromEntries(Object.keys(names).map((key) => [key, false])); + it("P1 each pass flag maps to exactly its pass id", () => { + assert.deepEqual([...canonicalEnabledPassIds(config({ reduction: { passes: allOff } }))], []); + for (const [flag, id] of Object.entries(names)) { + const cfg = config({ reduction: { passes: { ...allOff, [flag]: true } } }); + assert.deepEqual([...canonicalEnabledPassIds(cfg)], [id], flag); + } + }); + it("P2 maxToolChars feeds tool_payload_trim and passOptions override it", () => { + assert.equal(canonicalPassOptions(config({ reduction: { maxToolChars: 1800 } })).tool_payload_trim?.maxChars, 1800); + const cfg = config({ reduction: { maxToolChars: 1800, passOptions: { toolPayloadTrim: { maxChars: 999 } } } }); + assert.equal(canonicalPassOptions(cfg).tool_payload_trim?.maxChars, 999); + }); +}); + +describe("reduceCanonicalEnvelope", () => { + it("R1 reduction module off leaves the envelope untouched", async () => { + const env = envelope(toolTurn("c1", BIG)); + const out = await reduceCanonicalEnvelope({ envelope: env, config: config({ modules: { reduction: false } }) }); + assert.equal(out.envelope, env); + assert.equal(out.summary.skippedReason, "disabled"); + }); + it("R2 no tool results", async () => { + const env = envelope([{ role: "user", content: BIG }]); + const out = await reduceCanonicalEnvelope({ envelope: env, config: config() }); + assert.equal(out.envelope, env); + assert.equal(out.summary.skippedReason, "no_candidate_segments"); + }); + it("R3 candidates below triggerMinChars", async () => { + const env = envelope(toolTurn("c1", "x".repeat(500))); + const out = await reduceCanonicalEnvelope({ envelope: env, config: config() }); + assert.equal(out.envelope, env); + assert.equal(out.summary.skippedReason, "below_trigger_min_chars"); + }); + it("R4 all passes disabled has no effect", async () => { + const env = envelope(toolTurn("c1", BIG)); + const passes = { readStateCompaction: false, toolPayloadTrim: false, htmlSlimming: false, execOutputTruncation: false, agentsStartupOptimization: false }; + const out = await reduceCanonicalEnvelope({ envelope: env, config: config({ reduction: { passes } }) }); + assert.equal(out.envelope, env); + assert.equal(out.summary.skippedReason, "pipeline_no_effect"); + }); + it("R5 an applied reduction changes only tool_result text and never mutates the input", async () => { + const env = envelope([{ role: "user", content: "build" }, ...toolTurn("c1", BIG)]); + const snapshot = JSON.stringify(env); + const out = await reduceCanonicalEnvelope({ envelope: env, config: config() }); + assert.equal(JSON.stringify(env), snapshot, "input mutated"); + assert.notEqual(out.envelope, env); + assert.deepEqual(out.envelope.messages.slice(0, 2), env.messages.slice(0, 2)); + const after = resultText(out.envelope, 2); + assert.ok(after.length < BIG.length); + assert.equal(out.summary.changedBlocks, 1); + assert.equal(out.summary.savedChars, BIG.length - after.length); + assert.equal(out.summary.beforeChars - out.summary.afterChars, out.summary.savedChars); + assert.equal(out.summary.visualSegments?.[0]?.segmentId, "tool-c1"); + }); + it("R6 output text equals the shared pipeline run directly on the same inputs", async () => { + const env = envelope([{ role: "user", content: "build" }, ...toolTurn("c1", BIG), ...toolTurn("c2", JSON.stringify({ rows: Array.from({ length: 300 }, (_, i) => ({ i })) }), "read", { path: "/a.json" })]); + const cfg = config(); + const out = await reduceCanonicalEnvelope({ envelope: env, config: cfg }); + const inputs = prepareCanonicalReductionInputs(env, cfg); + const direct = await runReductionBeforeCall({ turnCtx: inputs.turnCtx, passes: inputs.passes }); + const directById = new Map(direct.turnCtx.segments.map((s) => [s.id, s.text])); + assert.equal(resultText(out.envelope, 2), directById.get("tool-c1")); + assert.equal(resultText(out.envelope, 4), directById.get("tool-c2")); + }); + it("R7 with the memo a second run is byte-identical", async () => { + const env = envelope([{ role: "user", content: "build" }, ...toolTurn("c1", BIG)], "s-memo"); + const memo = new ReductionMemo(); + const first = await reduceCanonicalEnvelope({ envelope: env, config: config(), memo }); + await new Promise((resolve) => setTimeout(resolve, 5)); + const second = await reduceCanonicalEnvelope({ envelope: env, config: config(), memo }); + assert.equal(JSON.stringify(second.envelope.messages), JSON.stringify(first.envelope.messages)); + }); + it("R8 without the memo a rerun is byte-identical too", async () => { + const env = envelope([{ role: "user", content: "build" }, ...toolTurn("c1", BIG)], "s-nomemo"); + const first = await reduceCanonicalEnvelope({ envelope: env, config: config() }); + await new Promise((resolve) => setTimeout(resolve, 5)); + const second = await reduceCanonicalEnvelope({ envelope: env, config: config() }); + assert.equal(resultText(second.envelope, 2), resultText(first.envelope, 2)); + }); + it("R9 trimmed content is recoverable through the shared recovery tool", async () => { + const env = envelope([{ role: "user", content: "build" }, ...toolTurn("c1", BIG)], "s-recover"); + const out = await reduceCanonicalEnvelope({ envelope: env, config: config() }); + const dataKey = /"dataKey":"([^"]+)"/.exec(resultText(out.envelope, 2))?.[1]; + assert.equal(dataKey, "segment:tool-c1"); + const recovered = await resolveMemoryFaultRecover({ dataKey: dataKey!, stateDir }); + assert.ok(recovered.text.includes("line 2999 output from build step ok")); + assert.equal(recovered.details.error, undefined); + }); + it("R10 non-text blocks and metadata are preserved", async () => { + const env = envelope([ + { role: "user", content: [{ type: "text", text: "look" }, { type: "image", mediaType: "image/png" }] }, + { role: "assistant", content: [{ type: "text", text: "ok" }, { type: "tool_call", toolCallId: "c1", toolName: "bash" }], metadata: { reasoning: ["think"] } }, + { role: "tool", content: [{ type: "tool_result", toolCallId: "c1", toolName: "bash", status: "error", text: BIG }], metadata: { piIndex: 2 } }, + ]); + const out = await reduceCanonicalEnvelope({ envelope: env, config: config() }); + assert.deepEqual(out.envelope.messages[0], env.messages[0]); + assert.deepEqual(out.envelope.messages[1], env.messages[1]); + assert.deepEqual(out.envelope.messages[2]?.metadata, { piIndex: 2 }); + const block = (out.envelope.messages[2]?.content as Array>)[0]; + assert.equal(block?.status, "error"); + assert.equal(block?.toolCallId, "c1"); + }); +}); + +describe("passSavedChars", () => { + const effect = (id: string, changed: boolean, savedChars: number) => ({ id, changed, beforeChars: 100, afterChars: 100 - savedChars, savedChars }); + const summary = (passEffects: ReturnType[]) => ({ + changedMessages: 0, changedBlocks: 0, savedChars: 0, beforeChars: 0, afterChars: 0, report: [], passEffects, + diagnostics: { messageCount: 0, toolLikeMessages: 0, candidateSegments: 0, candidateChars: 0 }, + }); + it("PS1 an undefined summary yields an empty record", () => { + assert.deepEqual(passSavedChars(undefined), {}); + }); + it("PS2 only changed passes with savings are listed", () => { + assert.deepEqual(passSavedChars(summary([effect("tool_payload_trim", true, 40), effect("html_slimming", false, 0), effect("read_state_compaction", true, 0)])), { tool_payload_trim: 40 }); + }); + it("PS3 repeated pass ids are summed", () => { + assert.deepEqual(passSavedChars(summary([effect("a", true, 10), effect("a", true, 5)])), { a: 15 }); + }); +}); + +describe("ReductionMemo disclosed read paths", () => { + const seg = (id: string, path?: string) => ({ id, kind: "volatile", text: "x", priority: 0, metadata: path ? { path } : {} }) as never; + const all = (...ids: string[]) => new Set(ids); + it("M1 nothing recorded carries nothing", () => { + assert.equal(new ReductionMemo().carriedDisclosedReadPaths(new Set()), undefined); + }); + it("M2 a path whose owner is still present is not carried", () => { + const memo = new ReductionMemo(); + memo.recordDisclosedReadPaths(["/repo/a.ts"], [seg("tool-1", "/repo/a.ts")], all("tool-1")); + assert.equal(memo.carriedDisclosedReadPaths(new Set(["tool-1"])), undefined); + }); + it("M3 a path whose owner has left the history is carried", () => { + const memo = new ReductionMemo(); + memo.recordDisclosedReadPaths(["/repo/a.ts"], [seg("tool-1", "/repo/a.ts")], all("tool-1")); + assert.deepEqual(memo.carriedDisclosedReadPaths(new Set(["tool-2"])), ["/repo/a.ts"]); + }); + it("M4 an unattributable path is always carried", () => { + const memo = new ReductionMemo(); + memo.recordDisclosedReadPaths(["/repo/b.ts"], [seg("tool-1", "/repo/a.ts")], all("tool-1")); + assert.deepEqual(memo.carriedDisclosedReadPaths(new Set(["tool-1"])), ["/repo/b.ts"]); + }); + it("M5 recording ignores junk, normalizes, and a known path keeps its owner", () => { + const memo = new ReductionMemo(); + memo.recordDisclosedReadPaths("nope", [seg("tool-1", "/a")], all("tool-1")); + memo.recordDisclosedReadPaths([42, "", " "], [seg("tool-1", "/a")], all("tool-1")); + assert.equal(memo.disclosedReadPaths, undefined); + memo.recordDisclosedReadPaths([" /Repo/A.ts "], [seg("tool-1", "/repo/a.ts"), seg("tool-2", "/repo/a.ts")], all("tool-1", "tool-2")); + memo.recordDisclosedReadPaths(["/repo/a.ts"], [seg("tool-3", "/repo/a.ts")], all("tool-3")); + assert.deepEqual(memo.disclosedReadPaths, ["/repo/a.ts"]); + assert.equal(memo.carriedDisclosedReadPaths(new Set(["tool-1"])), undefined); + assert.deepEqual(memo.carriedDisclosedReadPaths(new Set(["tool-2", "tool-3"])), ["/repo/a.ts"]); + }); + it("M6 only an eligible segment can own a path", () => { + const memo = new ReductionMemo(); + memo.recordDisclosedReadPaths(["/repo/a.ts"], [seg("write-1", "/repo/a.ts"), seg("read-1", "/repo/a.ts")], all("read-1")); + assert.equal(memo.carriedDisclosedReadPaths(new Set(["read-1"])), undefined); + assert.deepEqual(memo.carriedDisclosedReadPaths(new Set(["write-1"])), ["/repo/a.ts"]); + const none = new ReductionMemo(); + none.recordDisclosedReadPaths(["/repo/a.ts"], [seg("read-1", "/repo/a.ts")], all()); + assert.deepEqual(none.carriedDisclosedReadPaths(new Set(["read-1"])), ["/repo/a.ts"], "no eligible owner → unattributable"); + }); + it("M7 paths missing from the reported set are dropped", () => { + const memo = new ReductionMemo(); + memo.recordDisclosedReadPaths(["/a", "/b"], [seg("t1", "/a"), seg("t2", "/b")], all("t1", "t2")); + memo.recordDisclosedReadPaths(["/b"], [], all()); + assert.deepEqual(memo.disclosedReadPaths, ["/b"]); + assert.equal(memo.carriedDisclosedReadPaths(new Set(["t2"])), undefined, "/b keeps its owner"); + }); + it("M8 dirty tracks real changes only", () => { + const memo = new ReductionMemo(); + assert.equal(memo.dirty, false); + memo.recordDisclosedReadPaths(["/a"], [seg("t1", "/a")], all("t1")); + assert.equal(memo.dirty, true); + memo.dirty = false; + memo.recordDisclosedReadPaths(["/A "], [seg("t9", "/a")], all("t9")); + assert.equal(memo.dirty, false, "same set, same owner"); + memo.recordDisclosedReadPaths(["/a", "/b"], [], all()); + assert.equal(memo.dirty, true, "path added"); + memo.dirty = false; + memo.recordDisclosedReadPaths(["/b"], [], all()); + assert.equal(memo.dirty, true, "path dropped"); + }); + it("M9 clear forgets everything", () => { + const memo = new ReductionMemo(); + memo.recordDisclosedReadPaths(["/a"], [seg("t1", "/a")], all("t1")); + memo.clear(); + assert.equal(memo.disclosedReadPaths, undefined); + assert.equal(memo.dirty, false); + }); +}); + +describe("reduceCanonicalEnvelope across requests (progressive disclosure)", () => { + const CODE = Array.from({ length: 400 }, (_, i) => `export function handler${i}(input: string): string {\n const value = input.trim();\n return value + "${i}";\n}\n`).join("\n"); + const PATH = "/repo/src/handlers.ts"; + const user = (text: string): RuntimeMessage => ({ role: "user", content: [{ type: "text", text }] }); + const read = (callId: string) => toolTurn(callId, CODE, "read", { path: PATH }); + const trimmedAt = (env: HostRequestEnvelope, index: number) => resultText(env, index).length < CODE.length; + + it("D1 a trimmed read still in history stays trimmed and byte-identical on the next request", async () => { + const memo = new ReductionMemo(); + const first = [user("read the handlers"), ...read("r1")]; + const a = await reduceCanonicalEnvelope({ envelope: envelope(first, "d1"), config: config(), memo }); + assert.ok(trimmedAt(a.envelope, 2), "precondition: the first read is trimmed"); + const next = [...first, ...toolTurn("w1", "wrote 10 bytes", "write", { path: "/repo/out.txt" })]; + const b = await reduceCanonicalEnvelope({ envelope: envelope(next, "d1"), config: config(), memo }); + assert.equal(resultText(b.envelope, 2), resultText(a.envelope, 2)); + const c = await reduceCanonicalEnvelope({ envelope: envelope(next, "d1"), config: config(), memo }); + assert.equal(resultText(c.envelope, 2), resultText(a.envelope, 2)); + }); + + it("D2 after the disclosing read leaves history, a fresh read of the same path is left untrimmed", async () => { + const memo = new ReductionMemo(); + const a = await reduceCanonicalEnvelope({ envelope: envelope([user("read"), ...read("r1")], "d2"), config: config(), memo }); + assert.ok(trimmedAt(a.envelope, 2)); + const compacted = [user("summary of earlier work"), user("read it again in full"), ...read("r2")]; + const b = await reduceCanonicalEnvelope({ envelope: envelope(compacted, "d2"), config: config(), memo }); + assert.equal(resultText(b.envelope, 3), CODE); + }); + + it("D3 two reads of one path in one history: first trimmed, second untrimmed, stable across requests", async () => { + const memo = new ReductionMemo(); + const history = [user("read"), ...read("r1"), user("again"), ...read("r2")]; + const a = await reduceCanonicalEnvelope({ envelope: envelope(history, "d3"), config: config(), memo }); + assert.ok(trimmedAt(a.envelope, 2)); + assert.equal(resultText(a.envelope, 5), CODE); + const b = await reduceCanonicalEnvelope({ envelope: envelope(history, "d3"), config: config(), memo }); + assert.equal(resultText(b.envelope, 2), resultText(a.envelope, 2)); + assert.equal(resultText(b.envelope, 5), CODE); + }); + + it("D4 ownership follows the same-path read actually trimmed", async () => { + const memo = new ReductionMemo(); + const a = await reduceCanonicalEnvelope({ + envelope: envelope([user("read it twice"), ...toolTurn("short", "not found", "read", { path: PATH }), ...read("long")], "d4"), + config: config(), + memo, + }); + assert.ok(trimmedAt(a.envelope, 4), "precondition: only the long read is trimmed"); + assert.equal(resultText(a.envelope, 2), "not found"); + assert.equal(memo.carriedDisclosedReadPaths(new Set(["tool-long"])), undefined, "the trimmed read owns the path"); + const b = await reduceCanonicalEnvelope({ envelope: envelope([user("after compaction"), ...read("long")], "d4"), config: config(), memo }); + assert.ok(trimmedAt(b.envelope, 2), "the retained long read must not expand back to full content"); + }); + + it("D5 a write to the same path cannot own the disclosure", async () => { + const memo = new ReductionMemo(); + const history = [user("write then read"), ...toolTurn("w1", "wrote the file", "write", { path: PATH }), ...read("r1")]; + const a = await reduceCanonicalEnvelope({ envelope: envelope(history, "d5"), config: config(), memo }); + assert.ok(trimmedAt(a.envelope, 4)); + assert.equal(memo.carriedDisclosedReadPaths(new Set(["tool-r1"])), undefined, "the read owns the path"); + assert.deepEqual(memo.carriedDisclosedReadPaths(new Set(["tool-w1"])), [PATH], "the write does not"); + }); +}); diff --git a/components/adapters/opencode/tsconfig.json b/components/adapters/opencode/tsconfig.json new file mode 100644 index 00000000..19420936 --- /dev/null +++ b/components/adapters/opencode/tsconfig.json @@ -0,0 +1,14 @@ +{ + "extends": "../../../tsconfig.base.json", + "compilerOptions": { + "types": [ + "node" + ] + }, + "include": [ + "src/**/*.ts", + "scripts/**/*.ts", + "tests/**/*.ts", + "../shared/canonical/**/*.ts" + ] +} diff --git a/components/adapters/shared/canonical/before-call.ts b/components/adapters/shared/canonical/before-call.ts new file mode 100644 index 00000000..3a8ba137 --- /dev/null +++ b/components/adapters/shared/canonical/before-call.ts @@ -0,0 +1,209 @@ +/** + * Request-time pipeline for in-process canonical adapters (pi, OpenCode). + * + * Reduction runs through the shared `prepareObservedBeforeCall` (same helper the + * Claude Code gateway uses), so ux-effects, report aggregates and the browser + * visual are recorded identically. The stable-prefix step lives in each host's + * system-prompt hook (`applyCanonicalStablePrefix` below) because in-process hosts + * such as OpenCode expose the system prompt separately from the message history. + */ +import { + appendRecoveryProtocolText, + defaultInjectRecoveryProtocol, + type HostPayloadCodec, + type HostRequestEnvelope, + type HostSessionContext, +} from "@lightrsi/host-adapter"; +import type { RuntimeMessage } from "@lightrsi/kernel"; +import { + prepareObservedBeforeCall, + writeStabilityVisualSnapshot, +} from "@lightrsi/product-surface"; +import { + applyStablePrefixToInstructions, + buildStabilityVisualSnapshotFromEnvelopes, + rewriteTextForStablePrefix, +} from "@lightrsi/stabilizer"; +import type { CanonicalAdapterConfig } from "./config.js"; +import { reduceCanonicalEnvelope, type CanonicalReductionSummary, type ReductionMemo } from "./reduction.js"; + +/** Envelopes here are already canonical; the codec is the identity. */ +export const CANONICAL_IDENTITY_CODEC: HostPayloadCodec = { + decodeRequest(rawPayload) { + return rawPayload as HostRequestEnvelope; + }, + encodeRequest(envelope) { + return envelope; + }, + decodeResponse(rawResponse, request) { + return { request, rawResponse } as never; + }, + encodeResponse(_envelope, originalRawResponse) { + return originalRawResponse; + }, +}; + +export function createCanonicalEnvelope(params: { + hostId: string; + displayName: string; + sessionId: string; + turnId?: string; + model: string; + instructions?: string; + messages: RuntimeMessage[]; + metadata?: Record; +}): HostRequestEnvelope { + const session: HostSessionContext = { + host: { hostId: params.hostId, displayName: params.displayName }, + sessionId: params.sessionId, + ...(params.turnId ? { turnId: params.turnId } : {}), + sessionMode: "single", + }; + return { + session, + model: params.model, + stream: true, + ...(params.instructions !== undefined ? { instructions: params.instructions } : {}), + messages: params.messages, + rawPayload: undefined, + metadata: params.metadata, + }; +} + +export async function runCanonicalBeforeCallReduction(params: { + envelope: HostRequestEnvelope; + config: CanonicalAdapterConfig; + memo?: ReductionMemo; +}): Promise<{ messages: RuntimeMessage[]; summary?: CanonicalReductionSummary }> { + const { envelope, config } = params; + const sessionId = envelope.session.sessionId; + const prepared = await prepareObservedBeforeCall({ + envelope, + codec: CANONICAL_IDENTITY_CODEC, + config: { mode: "normal" }, + prepareStablePrefix: (next) => next, + applyBeforeCallReduction: ({ envelope: next }) => reduceCanonicalEnvelope({ envelope: next, config, memo: params.memo }), + observability: { + stateDir: config.stateDir, + sessionId, + model: envelope.model, + recordUxEffectNow: true, + buildReduction(summary) { + return summary.savedChars > 0 + ? { + countMode: "chars", + beforeCount: summary.beforeChars, + afterCount: summary.afterChars, + savedCount: summary.savedChars, + details: { requestSavedCount: summary.savedChars }, + segments: (summary.visualSegments ?? []).map((segment) => ({ + segmentId: segment.segmentId, + itemIndex: segment.messageIndex, + field: "content", + blockIndex: segment.blockIndex, + toolName: segment.toolName, + savedChars: segment.savedChars, + beforeText: segment.beforeText, + afterText: segment.afterText, + report: segment.report, + })), + } + : undefined; + }, + }, + }); + // The shared pipeline also appends the recovery protocol to `instructions`. + // In-process hosts own their system prompt through a separate hook, which injects + // it via `withCanonicalRecoveryProtocol`; only messages are returned here so the + // two paths cannot silently diverge. + return { messages: prepared.envelope.messages, summary: prepared.reductionSummary }; +} + +/** + * The shared recovery protocol text, obtained from the shared injector so it is + * never duplicated here. Empty when `TOKENPILOT_DISABLE_RECOVERY_PROTOCOL` is set. + */ +export function canonicalRecoveryProtocolText(): string { + const probe = defaultInjectRecoveryProtocol({ + session: { host: { hostId: "canonical", displayName: "canonical" }, sessionId: "", sessionMode: "single" }, + model: "", + stream: false, + instructions: "", + messages: [], + rawPayload: undefined, + }); + return typeof probe.instructions === "string" ? probe.instructions : ""; +} + +/** Append the shared recovery protocol to a system prompt once (idempotent). */ +export function withCanonicalRecoveryProtocol(systemText: string): { changed: boolean; text: string } { + const protocolText = canonicalRecoveryProtocolText(); + if (!protocolText) return { changed: false, text: systemText }; + const result = appendRecoveryProtocolText({ currentInstructions: systemText, protocolText }); + return { changed: result.changed, text: result.instructions }; +} + +export type CanonicalStablePrefixResult = { + changed: boolean; + /** System/instruction text to send. */ + instructions: string; + /** Volatile lines removed from the stable prefix. */ + dynamicContextText: string; + /** For target `user`: text to prepend to the first user message (request-local). */ + userPrefix?: string; +}; + +/** + * Apply the shared stabilizer to one system-prompt text. Target `developer` keeps + * the volatile lines in the system prompt but moves them to its tail + * (`mergeDynamicContextIntoInstructions`); target `user` removes them from the + * system prompt and returns them for the first user message. + */ +export function applyCanonicalStablePrefix( + instructions: string, + config: CanonicalAdapterConfig, +): CanonicalStablePrefixResult { + const unchanged = { changed: false, instructions, dynamicContextText: "" }; + if (!config.modules.stabilizer || !instructions.trim()) return unchanged; + const target = config.hooks.dynamicContextTarget; + const rewrite = rewriteTextForStablePrefix(instructions); + if (!rewrite.changed || !rewrite.dynamicContextText) return unchanged; + const probe = applyStablePrefixToInstructions({ + envelope: { + session: { host: { hostId: "canonical" } }, + model: "", + instructions, + messages: [{ role: "user", content: "" }], + }, + dynamicContextTarget: target, + mergeDynamicContextIntoInstructions: target === "developer", + }); + const next = typeof probe.instructions === "string" ? probe.instructions : instructions; + return { + changed: next !== instructions, + instructions: next, + dynamicContextText: rewrite.dynamicContextText, + ...(target === "user" ? { userPrefix: rewrite.dynamicContextText } : {}), + }; +} + +export async function recordCanonicalStability(params: { + config: CanonicalAdapterConfig; + sessionId: string; + model: string; + before: string; + result: CanonicalStablePrefixResult; +}): Promise { + if (!params.result.changed) return; + const snapshot = buildStabilityVisualSnapshotFromEnvelopes({ + sessionId: params.sessionId, + model: params.model, + upstreamModel: params.model, + originalEnvelope: { instructions: params.before, messages: [] }, + preparedEnvelope: { instructions: params.result.instructions, messages: [] }, + dynamicContextTarget: params.config.hooks.dynamicContextTarget, + dynamicContextText: params.result.dynamicContextText, + getDeveloperText: (envelope) => (typeof envelope.instructions === "string" ? envelope.instructions : ""), + }); + await writeStabilityVisualSnapshot({ stateDir: params.config.stateDir, snapshot }); +} diff --git a/components/adapters/shared/canonical/config.ts b/components/adapters/shared/canonical/config.ts new file mode 100644 index 00000000..977c8240 --- /dev/null +++ b/components/adapters/shared/canonical/config.ts @@ -0,0 +1,213 @@ +/** + * Runtime config shared by the in-process canonical-transcript adapters + * (OpenCode). Keys and defaults match the Claude Code adapter + * (`claude-code/src/config.ts`) so the shared CLI (`mode`, `reduction`, + * `stabilizer`, `eviction`) edits the same shape on every host. Host-specific + * paths are supplied by each adapter. + */ +import { existsSync } from "node:fs"; +import { mkdir, readFile, rename, writeFile } from "node:fs/promises"; +import { homedir } from "node:os"; +import { dirname, join } from "node:path"; + +export type CanonicalReductionPassName = + | "readStateCompaction" + | "toolPayloadTrim" + | "htmlSlimming" + | "execOutputTruncation" + | "agentsStartupOptimization"; + +export const CANONICAL_REDUCTION_PASS_NAMES: readonly CanonicalReductionPassName[] = [ + "readStateCompaction", + "toolPayloadTrim", + "htmlSlimming", + "execOutputTruncation", + "agentsStartupOptimization", +]; + +export type CanonicalAdapterConfig = { + enabled: boolean; + logLevel: "info" | "debug"; + stateDir: string; + hooks: { + dynamicContextTarget: "developer" | "user"; + }; + modules: { + stabilizer: boolean; + reduction: boolean; + eviction: boolean; + }; + eviction: { + enabled: boolean; + minBlockChars: number; + failureMode: "bypass"; + }; + reduction: { + triggerMinChars: number; + maxToolChars: number; + passes: Record; + passOptions: Record>; + }; + taskStateEstimator: { + enabled?: boolean; + baseUrl?: string; + apiKey?: string; + model?: string; + requestTimeoutMs: number; + batchTurns: number; + evictionLookaheadTurns: number; + inputMode?: "sliding_window" | "completed_summary_plus_active_turns"; + lifecycleMode?: "coupled" | "decoupled"; + evidenceMode?: "two_state" | "three_state"; + }; +}; + +export function runtimeHomeDir(): string { + return process.env.HOME?.trim() || process.env.USERPROFILE?.trim() || homedir(); +} + +export function expandHomePath(value: string): string { + if (value === "~") return runtimeHomeDir(); + if (value.startsWith("~/")) return join(runtimeHomeDir(), value.slice(2)); + return value; +} + +function asRecord(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) + ? value as Record + : {}; +} + +function boolValue(value: unknown, fallback: boolean): boolean { + return typeof value === "boolean" ? value : fallback; +} + +function numberValue(value: unknown, fallback: number, min: number, max: number): number { + const next = typeof value === "number" ? value : Number(value); + if (!Number.isFinite(next)) return fallback; + return Math.max(min, Math.min(max, Math.trunc(next))); +} + +function stringValue(value: unknown): string | undefined { + return typeof value === "string" && value.trim() ? value.trim() : undefined; +} + +function sanitizePassOptions(raw: unknown): Record> { + const input = asRecord(raw); + const output: Record> = {}; + for (const key of CANONICAL_REDUCTION_PASS_NAMES) { + const value = asRecord(input[key]); + if (Object.keys(value).length > 0) output[key] = value; + } + return output; +} + +export function normalizeCanonicalAdapterConfig( + raw: unknown, + options: { defaultStateDir: string }, +): CanonicalAdapterConfig { + const obj = asRecord(raw); + const hooks = asRecord(obj.hooks); + const modules = asRecord(obj.modules); + const eviction = asRecord(obj.eviction); + const reduction = asRecord(obj.reduction); + const passes = asRecord(reduction.passes); + const estimator = asRecord(obj.taskStateEstimator); + return { + enabled: boolValue(obj.enabled, true), + logLevel: obj.logLevel === "debug" ? "debug" : "info", + stateDir: expandHomePath(stringValue(obj.stateDir) ?? options.defaultStateDir), + hooks: { + dynamicContextTarget: hooks.dynamicContextTarget === "user" ? "user" : "developer", + }, + modules: { + stabilizer: boolValue(modules.stabilizer, true), + reduction: boolValue(modules.reduction, true), + eviction: boolValue(modules.eviction, false), + }, + eviction: { + enabled: boolValue(eviction.enabled, false), + minBlockChars: numberValue(eviction.minBlockChars, 4000, 256, 1_000_000), + failureMode: "bypass", + }, + reduction: { + triggerMinChars: numberValue(reduction.triggerMinChars, 2200, 256, 1_000_000), + maxToolChars: numberValue(reduction.maxToolChars, 1200, 256, 1_000_000), + passes: { + readStateCompaction: boolValue(passes.readStateCompaction, true), + toolPayloadTrim: boolValue(passes.toolPayloadTrim, true), + htmlSlimming: boolValue(passes.htmlSlimming, true), + execOutputTruncation: boolValue(passes.execOutputTruncation, true), + agentsStartupOptimization: boolValue(passes.agentsStartupOptimization, true), + }, + passOptions: sanitizePassOptions(reduction.passOptions), + }, + taskStateEstimator: { + enabled: typeof estimator.enabled === "boolean" ? estimator.enabled : undefined, + baseUrl: stringValue(estimator.baseUrl), + apiKey: stringValue(estimator.apiKey), + model: stringValue(estimator.model), + requestTimeoutMs: numberValue(estimator.requestTimeoutMs, 60_000, 1000, 600_000), + batchTurns: numberValue(estimator.batchTurns, 5, 1, 1000), + evictionLookaheadTurns: numberValue(estimator.evictionLookaheadTurns, 3, 1, 1000), + inputMode: estimator.inputMode === "sliding_window" + ? "sliding_window" + : estimator.inputMode === "completed_summary_plus_active_turns" + ? "completed_summary_plus_active_turns" + : undefined, + lifecycleMode: estimator.lifecycleMode === "decoupled" + ? "decoupled" + : estimator.lifecycleMode === "coupled" ? "coupled" : undefined, + evidenceMode: estimator.evidenceMode === "two_state" + ? "two_state" + : estimator.evidenceMode === "three_state" ? "three_state" : undefined, + }, + }; +} + +export async function loadCanonicalAdapterConfig( + configPath: string, + options: { defaultStateDir: string }, +): Promise { + if (!existsSync(configPath)) return normalizeCanonicalAdapterConfig({}, options); + const text = await readFile(configPath, "utf8"); + return normalizeCanonicalAdapterConfig(JSON.parse(text), options); +} + +export async function writeCanonicalAdapterConfig(config: CanonicalAdapterConfig, configPath: string): Promise { + await mkdir(dirname(configPath), { recursive: true }); + const tempPath = `${configPath}.tmp`; + await writeFile(tempPath, `${JSON.stringify(config, null, 2)}\n`, { encoding: "utf8", mode: 0o600 }); + await rename(tempPath, configPath); +} + +export type EvictionReadiness = { + /** True only when the host declares eviction and every setting is complete. */ + active: boolean; + reason: + | "ready" + | "unsupported_on_host" + | "module_disabled" + | "eviction_disabled" + | "estimator_disabled" + | "estimator_incomplete"; + missing: string[]; +}; + +/** + * Eviction is opt-in. It runs only when the host binding declares it, both + * `modules.eviction` and `eviction.enabled` are on, the estimator is not + * explicitly disabled, and `taskStateEstimator.{baseUrl,model,apiKey}` are set. + */ +export function evaluateEvictionReadiness( + config: CanonicalAdapterConfig, + hostSupportsEviction: boolean, +): EvictionReadiness { + const missing = (["baseUrl", "model", "apiKey"] as const).filter((key) => !config.taskStateEstimator[key]); + if (!hostSupportsEviction) return { active: false, reason: "unsupported_on_host", missing }; + if (!config.modules.eviction) return { active: false, reason: "module_disabled", missing }; + if (!config.eviction.enabled) return { active: false, reason: "eviction_disabled", missing }; + if (config.taskStateEstimator.enabled === false) return { active: false, reason: "estimator_disabled", missing }; + if (missing.length > 0) return { active: false, reason: "estimator_incomplete", missing }; + return { active: true, reason: "ready", missing }; +} diff --git a/components/adapters/shared/canonical/eviction.ts b/components/adapters/shared/canonical/eviction.ts new file mode 100644 index 00000000..70db239e --- /dev/null +++ b/components/adapters/shared/canonical/eviction.ts @@ -0,0 +1,281 @@ +/** + * Canonical-surface eviction glue for transcript-based in-process hosts. + * + * The host supplies its model-visible surface as ordered `CanonicalSurfaceEntry` + * records (canonical message + stable entry id + user-turn number). This module + * builds the shared estimator snapshot and surface items, runs the shared + * `runCanonicalEvictionCycle` from `@lightrsi/eviction` (estimator → registry → + * safety policy), and hands the host a replacement per evicted entry. Each + * replacement archives the original through the shared artifact store so + * `memory_fault_recover` can restore it. + * + * Applying the replacement is host-specific: OpenCode re-applies a durable overlay + * request-locally; a host with native context edits can apply them directly. + */ +import { archiveContent, buildRecoveryHint } from "@lightrsi/artifact-store"; +import { + buildToolPairsFromItems, + createConfiguredTaskStateEstimator, + runCanonicalEvictionCycle, + type CanonicalEvictionCycleResult, + type SurfaceCycleItem, + type SurfaceTransactionOutcome, + type TaskStateEstimator, +} from "@lightrsi/eviction"; +import { + loadSessionTaskRegistry, + persistSessionTaskRegistry, + type RawSemanticMessageRecord, + type RawSemanticSnapshot, + type RawSemanticToolCallRecord, + type RawSemanticToolResultRecord, + type SessionTaskRegistry, + type TurnAnchor, +} from "@lightrsi/history"; +import type { RuntimeMessage } from "@lightrsi/kernel"; +import type { CanonicalAdapterConfig } from "./config.js"; + +export type CanonicalSurfaceEntry = { + /** Host-stable id of the entry (for OpenCode, the message or part id). */ + id: string; + /** User-turn number (1-based; turn N starts at the Nth user message). */ + turn: number; + message: RuntimeMessage; + /** Compaction/branch summaries: never evicted. */ + checkpoint?: boolean; + /** Already replaced by an earlier eviction: never evicted again. */ + alreadyEvicted?: boolean; +}; + +export type EvictionReplacement = { + id: string; + kind: string; + /** Model-visible replacement text (stub + recovery hint). */ + text: string; + originalChars: number; + dataKey: string; +}; + +const SUMMARY_MAX_CHARS = 600; + +function truncate(text: string): string { + const trimmed = text.trim(); + return trimmed.length <= SUMMARY_MAX_CHARS ? trimmed : `${trimmed.slice(0, SUMMARY_MAX_CHARS)}...`; +} + +function anchor(sessionId: string, turnSeq: number, role: TurnAnchor["role"]): TurnAnchor { + return { sessionId, turnAbsId: `${sessionId}:t${turnSeq}`, turnSeq, role }; +} + +export function canonicalVisibleText(message: RuntimeMessage): string { + if (typeof message.content === "string") return message.content; + return message.content + .map((block) => (block.type === "text" || block.type === "tool_result" ? block.text : "")) + .filter(Boolean) + .join("\n"); +} + +function toolCallIdsOf(message: RuntimeMessage): string[] { + if (!Array.isArray(message.content)) return []; + return message.content.flatMap((block) => (block.type === "tool_call" && block.toolCallId ? [block.toolCallId] : [])); +} + +function toolResultIdsOf(message: RuntimeMessage): string[] { + if (!Array.isArray(message.content)) return []; + return message.content.flatMap((block) => (block.type === "tool_result" && block.toolCallId ? [block.toolCallId] : [])); +} + +export function surfaceItemKind(entry: CanonicalSurfaceEntry): string { + if (entry.checkpoint) return "compaction_checkpoint"; + if (entry.message.role === "tool") return "tool_result"; + if (entry.message.role === "assistant" && toolCallIdsOf(entry.message).length > 0) return "tool_call"; + return "message"; +} + +/** Estimator snapshot over the current surface (shared `RawSemanticSnapshot`). */ +export function buildCanonicalRawSemanticSnapshot( + sessionId: string, + entries: readonly CanonicalSurfaceEntry[], +): RawSemanticSnapshot { + const messages: RawSemanticMessageRecord[] = []; + const toolCalls: RawSemanticToolCallRecord[] = []; + const toolResults: RawSemanticToolResultRecord[] = []; + const toolNameByCallId = new Map(); + let lastTurnSeq = 0; + + for (const entry of entries) { + lastTurnSeq = Math.max(lastTurnSeq, entry.turn); + const { message } = entry; + if (message.role === "user" || message.role === "assistant") { + const textBlocks = typeof message.content === "string" + ? message.content + : message.content.map((block) => (block.type === "text" ? block.text : "")).filter(Boolean).join("\n"); + if (textBlocks.trim()) { + messages.push({ anchor: anchor(sessionId, entry.turn, message.role), role: message.role, text: textBlocks.trim() }); + } + if (message.role === "assistant" && Array.isArray(message.content)) { + for (const block of message.content) { + if (block.type !== "tool_call") continue; + const args = block.argumentsText ?? (block.argumentsJson ? JSON.stringify(block.argumentsJson) : ""); + toolNameByCallId.set(block.toolCallId, block.toolName); + toolCalls.push({ + anchor: anchor(sessionId, entry.turn, "assistant"), + toolCallId: block.toolCallId, + toolName: block.toolName, + argumentsText: args, + argumentsSummary: truncate(args), + }); + } + } + } else if (message.role === "tool" && Array.isArray(message.content)) { + for (const block of message.content) { + if (block.type !== "tool_result") continue; + const callId = block.toolCallId ?? ""; + toolResults.push({ + anchor: anchor(sessionId, entry.turn, "tool"), + toolCallId: callId, + toolName: block.toolName ?? toolNameByCallId.get(callId) ?? "", + status: block.status === "error" ? "error" : "success", + fullText: block.text, + summary: truncate(block.text), + }); + } + } + } + return { sessionId, lastTurnSeq, messages, toolCalls, toolResults }; +} + +export function buildCanonicalSurfaceItems(entries: readonly CanonicalSurfaceEntry[]): SurfaceCycleItem[] { + return entries.map((entry) => { + const kind = surfaceItemKind(entry); + const callIds = kind === "tool_call" ? toolCallIdsOf(entry.message) : kind === "tool_result" ? toolResultIdsOf(entry.message) : []; + return { + sourceEventSeq: entry.id, + turn: entry.turn, + kind, + ...(callIds.length > 0 ? { callIds } : {}), + chars: canonicalVisibleText(entry.message).length, + }; + }); +} + +/** + * Archive the original and build the stub the model sees instead. Throws when + * the archive cannot be written: an eviction without a recoverable original is + * never applied. + */ +export async function buildEvictionReplacement(params: { + sessionId: string; + entry: CanonicalSurfaceEntry; +}): Promise { + const kind = surfaceItemKind(params.entry); + const originalText = canonicalVisibleText(params.entry.message); + const dataKey = `evicted:${params.entry.id}`; + const segmentId = `evict-${params.entry.id}`; + const toolName = Array.isArray(params.entry.message.content) + ? params.entry.message.content.find((block) => block.type === "tool_result")?.toolName + : undefined; + let hint = ""; + if (originalText.length > 0) { + const location = await archiveContent({ + sessionId: params.sessionId, + segmentId, + sourcePass: "lifecycle_eviction", + toolName: toolName ?? kind, + dataKey, + originalText, + metadata: { entryId: params.entry.id, kind, turn: params.entry.turn }, + }); + hint = buildRecoveryHint({ + dataKey, + originalSize: originalText.length, + archivePath: location.archivePath, + sourceLabel: "Evicted completed-task context", + }); + } + return { + id: params.entry.id, + kind, + text: `[evicted: ${kind} ${params.entry.id}]${hint}`, + originalChars: originalText.length, + dataKey, + }; +} + +export function createCanonicalEstimator(config: CanonicalAdapterConfig): TaskStateEstimator | undefined { + return createConfiguredTaskStateEstimator({ + enabled: config.taskStateEstimator.enabled, + baseUrl: config.taskStateEstimator.baseUrl, + apiKey: config.taskStateEstimator.apiKey, + model: config.taskStateEstimator.model, + requestTimeoutMs: config.taskStateEstimator.requestTimeoutMs, + batchTurns: config.taskStateEstimator.batchTurns, + evictionLookaheadTurns: config.taskStateEstimator.evictionLookaheadTurns, + inputMode: config.taskStateEstimator.inputMode, + lifecycleMode: config.taskStateEstimator.lifecycleMode, + evidenceMode: config.taskStateEstimator.evidenceMode, + }); +} + +export type CanonicalEvictionOutcome = SurfaceTransactionOutcome & { + replacements: EvictionReplacement[]; +}; + +/** + * One eviction cycle over the host's current surface. `apply` receives the + * archived replacements and must report whether they all landed. + */ +export async function runCanonicalSurfaceEviction(params: { + sessionId: string; + entries: readonly CanonicalSurfaceEntry[]; + config: CanonicalAdapterConfig; + estimator: TaskStateEstimator; + apply(replacements: EvictionReplacement[]): Promise | SurfaceTransactionOutcome["status"]; + registryStore?: { + load(sessionId: string): Promise | SessionTaskRegistry; + persist(registry: SessionTaskRegistry, expectedVersion: number): Promise | void; + }; +}): Promise> { + const { sessionId, entries, config } = params; + const store = params.registryStore ?? { + load: (id: string) => loadSessionTaskRegistry(config.stateDir, id), + persist: async (registry: SessionTaskRegistry, expectedVersion: number) => { + await persistSessionTaskRegistry(config.stateDir, registry, { expectedVersion }); + }, + }; + const registry = await store.load(sessionId); + const effective = entries.filter((entry) => !entry.alreadyEvicted); + const items = buildCanonicalSurfaceItems(effective); + const effectiveIds = items.map((item) => item.sourceEventSeq); + const byId = new Map(effective.map((entry) => [entry.id, entry])); + const currentTurn = entries.reduce((max, entry) => Math.max(max, entry.turn), 0); + + return runCanonicalEvictionCycle({ + snapshot: buildCanonicalRawSemanticSnapshot(sessionId, entries), + registry, + estimator: params.estimator, + items, + effectiveSeqs: effectiveIds, + pairs: buildToolPairsFromItems(items, effectiveIds), + currentTurn, + minBlockChars: config.eviction.minBlockChars, + allowSurfaceMutation: config.eviction.enabled, + persistRegistry: (next, expectedVersion) => store.persist(next, expectedVersion), + emptyResult: () => ({ status: "empty", replacements: [] }), + async apply(evictIds) { + const replacements: EvictionReplacement[] = []; + for (const id of evictIds) { + const entry = byId.get(id); + if (!entry) return { status: "deferred", replacements: [] }; + try { + replacements.push(await buildEvictionReplacement({ sessionId, entry })); + } catch { + // No recoverable archive → no eviction for this batch (watermark stays). + return { status: "deferred", replacements: [] }; + } + } + const status = await params.apply(replacements); + return { status, replacements: status === "committed" ? replacements : [] }; + }, + }); +} diff --git a/components/adapters/shared/canonical/logger.ts b/components/adapters/shared/canonical/logger.ts new file mode 100644 index 00000000..05117da6 --- /dev/null +++ b/components/adapters/shared/canonical/logger.ts @@ -0,0 +1,69 @@ +/** + * Fail-open guard and file logger for in-process adapters. + * + * In-process hosts own the terminal (for example OpenCode's TUI), so adapter + * diagnostics go to `/tokenpilot/adapter.log`, never stdout/stderr. + * `failOpen` runs a hook body and, on any error, logs it and returns the + * host-neutral fallback (leave the request unmodified). + */ +import { appendFileSync, mkdirSync } from "node:fs"; +import { dirname, join } from "node:path"; + +export type AdapterLogger = { + info(message: string, detail?: unknown): void; + debug(message: string, detail?: unknown): void; + warn(message: string, detail?: unknown): void; +}; + +export function adapterLogPath(stateDir: string): string { + return join(stateDir, "tokenpilot", "adapter.log"); +} + +function formatDetail(detail: unknown): string { + if (detail === undefined) return ""; + if (detail instanceof Error) return ` ${detail.name}: ${detail.message}`; + try { + return ` ${JSON.stringify(detail)}`; + } catch { + return ` ${String(detail)}`; + } +} + +export function createFileLogger(params: { + hostId: string; + stateDir: () => string | undefined; + debug: () => boolean; +}): AdapterLogger { + const write = (level: string, message: string, detail?: unknown) => { + const stateDir = params.stateDir(); + if (!stateDir) return; + try { + const path = adapterLogPath(stateDir); + mkdirSync(dirname(path), { recursive: true }); + appendFileSync(path, `${new Date().toISOString()} ${level} [${params.hostId}] ${message}${formatDetail(detail)}\n`, "utf8"); + } catch { + // Logging is best-effort; never let it affect the host. + } + }; + return { + info: (message, detail) => write("info", message, detail), + warn: (message, detail) => write("warn", message, detail), + debug: (message, detail) => { + if (params.debug()) write("debug", message, detail); + }, + }; +} + +export async function failOpen( + logger: AdapterLogger, + hook: string, + body: () => Promise | T, + fallback: T, +): Promise { + try { + return await body(); + } catch (error) { + logger.warn(`${hook} failed open`, error); + return fallback; + } +} diff --git a/components/adapters/shared/canonical/memo-store.ts b/components/adapters/shared/canonical/memo-store.ts new file mode 100644 index 00000000..eed85382 --- /dev/null +++ b/components/adapters/shared/canonical/memo-store.ts @@ -0,0 +1,32 @@ +/** + * Per-session persistence for the disclosed-read memo (`ReductionMemo`). + * + * Stored at `/tokenpilot/reduction-memo/.json`, so a host restart + * (for example `opencode run --continue`) keeps knowing which read disclosed which + * file. Loading never throws: a missing or unreadable file gives an empty memo, which + * at worst trims one deliberate re-read of a file whose first read was compacted away. + */ +import { readFile } from "node:fs/promises"; +import { join } from "node:path"; +import { writeJsonFileAtomic } from "@lightrsi/host-adapter"; +import { ReductionMemo } from "./reduction.js"; + +export function reductionMemoPath(stateDir: string, sessionId: string): string { + return join(stateDir, "tokenpilot", "reduction-memo", `${encodeURIComponent(sessionId)}.json`); +} + +export async function loadReductionMemo(stateDir: string, sessionId: string): Promise { + try { + return ReductionMemo.fromSnapshot(JSON.parse(await readFile(reductionMemoPath(stateDir, sessionId), "utf8"))); + } catch { + return new ReductionMemo(); + } +} + +/** Write the memo if it changed since it was loaded or last saved. Returns whether it wrote. */ +export async function saveReductionMemoIfDirty(stateDir: string, sessionId: string, memo: ReductionMemo): Promise { + if (!memo.dirty) return false; + await writeJsonFileAtomic(reductionMemoPath(stateDir, sessionId), memo.toSnapshot()); + memo.dirty = false; + return true; +} diff --git a/components/adapters/shared/canonical/reduction.ts b/components/adapters/shared/canonical/reduction.ts new file mode 100644 index 00000000..e97a5d0c --- /dev/null +++ b/components/adapters/shared/canonical/reduction.ts @@ -0,0 +1,688 @@ +/** + * Before-call reduction over a canonical `HostRequestEnvelope`. + * + * Mirrors the Claude Code adapter's `reduction.ts` (segment metadata, analyzer + + * fallback instructions, enabled-pass selection) but reads canonical + * `tool_result` blocks instead of Anthropic payload blocks, so in-process hosts + * (OpenCode) share one implementation. All reduction decisions and rewrites come + * from `@lightrsi/reduction` (`analyze*`, `resolveReductionPasses`, + * `runReductionBeforeCall`); nothing here changes what a pass does. + * + * The one adapter-side addition is `ReductionMemo`, which tracks which read + * disclosed each file path so the pass's repeat-read rule is not applied to the + * same read when the host resends the whole history (see + * docs/adapters/opencode-design.md). + */ +import type { HostRequestEnvelope } from "@lightrsi/host-adapter"; +import type { ContextSegment, RuntimeContentBlock, RuntimeMessage, RuntimeTurnContext } from "@lightrsi/kernel"; +import { + analyzeExecOutputTruncation, + analyzeToolPayloadTrim, + resolveReductionPasses, + runReductionBeforeCall, +} from "@lightrsi/reduction"; +import type { CanonicalAdapterConfig } from "./config.js"; + +type ReductionInstruction = { + strategy: string; + segmentIds: string[]; + confidence: number; + priority: number; + rationale: string; + parameters?: Record; +}; + +export type CanonicalReductionReportEntry = { + id: string; + phase: string; + target: string; + changed: boolean; + skippedReason?: string; + note?: string; + beforeChars: number; + afterChars: number; + touchedSegmentIds?: string[]; +}; + +export type CanonicalReductionPassEffect = { + id: string; + changed: boolean; + skippedReason?: string; + beforeChars: number; + afterChars: number; + savedChars: number; + touchedSegmentIds?: string[]; +}; + +export type CanonicalReductionVisualSegment = { + segmentId: string; + messageIndex: number; + blockIndex: number; + toolName?: string; + savedChars: number; + beforeText: string; + afterText: string; + report: CanonicalReductionReportEntry[]; +}; + +export type CanonicalReductionSummary = { + changedMessages: number; + changedBlocks: number; + savedChars: number; + beforeChars: number; + afterChars: number; + report: CanonicalReductionReportEntry[]; + passEffects: CanonicalReductionPassEffect[]; + diagnostics: { + messageCount: number; + toolLikeMessages: number; + candidateSegments: number; + candidateChars: number; + }; + visualSegments?: CanonicalReductionVisualSegment[]; + skippedReason?: string; +}; + +type SegmentBinding = { + segmentId: string; + messageIndex: number; + blockIndex: number; + toolName?: string; +}; + +/* ------------------------------------------------------------------ * + * Disclosed-read memo + * ------------------------------------------------------------------ */ + +function normalizeDisclosedPath(value: unknown): string | undefined { + if (typeof value !== "string") return undefined; + const normalized = value.trim().toLowerCase(); + return normalized || undefined; +} + +export type ReductionMemoSnapshot = { + version: 2; + disclosed: Array<[path: string, segmentId: string | null]>; +}; + +/** + * Per-session record of which read disclosed each file path. + * + * `tool_payload_trim` sends a file read untrimmed when its path was already + * disclosed (the model re-reading a file it saw summarised). The host resends the + * whole history every request, so the disclosing read is usually still there and + * the pass re-detects it itself; carrying its path as well would make the pass + * treat that same read as a repeat and send it in full. The memo therefore carries + * a path only once its disclosing read has left the history (compaction, eviction, + * pruning). It is persisted per session (memo-store.ts) so host restarts keep it. + */ +export class ReductionMemo { + /** Normalized read path -> id of the segment that disclosed it (`undefined` if not attributable). */ + private disclosedBy = new Map(); + /** True when the memo changed since it was last loaded or saved. */ + dirty = false; + + toSnapshot(): ReductionMemoSnapshot { + return { + version: 2, + disclosed: [...this.disclosedBy].map(([path, segmentId]) => [path, segmentId ?? null]), + }; + } + + /** Rebuild from a snapshot. Anything malformed is skipped; the result is never dirty. */ + static fromSnapshot(raw: unknown): ReductionMemo { + const memo = new ReductionMemo(); + const snapshot = raw && typeof raw === "object" ? raw as { version?: unknown; disclosed?: unknown } : undefined; + if (snapshot?.version !== 2) return memo; + for (const entry of Array.isArray(snapshot.disclosed) ? snapshot.disclosed : []) { + if (!Array.isArray(entry)) continue; + const path = normalizeDisclosedPath(entry[0]); + if (path) memo.disclosedBy.set(path, typeof entry[1] === "string" ? entry[1] : undefined); + } + return memo; + } + + /** Every read path currently recorded as disclosed. */ + get disclosedReadPaths(): string[] | undefined { + return this.disclosedBy.size > 0 ? [...this.disclosedBy.keys()] : undefined; + } + + /** Paths to hand the pass as "already disclosed": only those whose disclosing read has left the history. */ + carriedDisclosedReadPaths(presentSegmentIds: ReadonlySet): string[] | undefined { + const carried = [...this.disclosedBy] + .filter(([, segmentId]) => segmentId === undefined || !presentSegmentIds.has(segmentId)) + .map(([path]) => path); + return carried.length > 0 ? carried : undefined; + } + + /** + * Rebuild the record from the pass's reported (bounded) path set. A path already + * recorded keeps its owner; a new path is owned by the first segment in + * `eligibleOwnerIds` with that path (a read actually trimmed in this request). + */ + recordDisclosedReadPaths( + paths: unknown, + segments: readonly ContextSegment[], + eligibleOwnerIds: ReadonlySet, + ): void { + if (!Array.isArray(paths)) return; + const next = new Map(); + for (const entry of paths) { + const path = normalizeDisclosedPath(entry); + if (!path || next.has(path)) continue; + if (this.disclosedBy.has(path)) { + next.set(path, this.disclosedBy.get(path)); + continue; + } + const owner = segments.find((segment) => eligibleOwnerIds.has(segment.id) && normalizeDisclosedPath(segment.metadata?.path) === path); + next.set(path, owner?.id); + } + const unchanged = next.size === this.disclosedBy.size + && [...next].every(([path, owner]) => this.disclosedBy.has(path) && this.disclosedBy.get(path) === owner); + if (unchanged) return; + this.disclosedBy = next; + this.dirty = true; + } + + clear(): void { + this.disclosedBy.clear(); + this.dirty = false; + } +} + +function isReadTool(toolName: string | undefined): boolean { + const normalized = toolName?.trim().toLowerCase(); + return normalized === "read" || normalized === "file_read"; +} + +/* ------------------------------------------------------------------ * + * Segment building (mirrors claude-code/src/reduction.ts) + * ------------------------------------------------------------------ */ + +function asRecord(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) + ? value as Record + : {}; +} + +function extractPathHint(value: unknown): string | undefined { + const record = asRecord(value); + for (const candidate of [record.path, record.file_path, record.filePath, record.filename]) { + if (typeof candidate !== "string") continue; + const trimmed = candidate.trim(); + if (trimmed) return trimmed; + } + return undefined; +} + +function isHtmlLikeText(text: string): boolean { + return /<\/?(html|body|main|section|article|nav|script|style|div|span|a|p|ul|ol|li|table|meta|link)\b/i.test(text); +} + +function looksLikeWebPayload(text: string): boolean { + if (isHtmlLikeText(text)) return true; + return /\b( 0 ? 1 : 0; +} + +function payloadKindForText(text: string): "stdout" | "stderr" | "json" | "blob" { + const trimmed = text.trim(); + if ((trimmed.startsWith("{") && trimmed.endsWith("}")) || (trimmed.startsWith("[") && trimmed.endsWith("]"))) { + try { + JSON.parse(trimmed); + return "json"; + } catch { + return "stdout"; + } + } + if (/(\bstderr\b|error:|traceback|exception)/i.test(text)) return "stderr"; + if (isHtmlLikeText(text)) return "blob"; + if (text.length > 500 && countLines(text) <= 2) return "blob"; + return "stdout"; +} + +function thresholdForTool(toolName: string, config: CanonicalAdapterConfig): number { + const normalized = toolName.toLowerCase(); + const optionThresholds = config.reduction.passOptions.execOutputTruncation?.toolThresholds; + if (optionThresholds && typeof optionThresholds === "object" && normalized in optionThresholds) { + const value = Number((optionThresholds as Record)[normalized]); + if (Number.isFinite(value) && value > 0) return value; + } + if (normalized === "bash" || normalized === "shell" || normalized === "powershell") return 30_000; + if (normalized === "grep" || normalized === "rg") return 20_000; + if (normalized === "read" || normalized === "file_read") return Infinity; + return 50_000; +} + +function segmentForText(params: { + id: string; + text: string; + toolName?: string; + latestUserQuery: string; + path?: string; +}): ContextSegment { + const payloadKind = payloadKindForText(params.text); + return { + id: params.id, + kind: "volatile", + text: params.text, + priority: 30, + source: "canonical.tool_result", + metadata: { + role: "tool", + isToolPayload: true, + payloadKind, + latestUserQuery: params.latestUserQuery, + ...(params.path ? { path: params.path } : {}), + toolPayload: { + enabled: true, + kind: payloadKind, + toolName: params.toolName ?? "tool", + ...(params.path ? { path: params.path } : {}), + }, + reduction: { + target: "tool_payload", + payloadKind, + toolPayloadTrim: { enabled: true, kind: payloadKind }, + }, + }, + }; +} + +function blockText(content: RuntimeMessage["content"]): string { + if (typeof content === "string") return content; + return content + .map((block) => (block.type === "text" || block.type === "tool_result" ? block.text : "")) + .filter(Boolean) + .join("\n"); +} + +function latestUserQuery(messages: RuntimeMessage[]): string { + for (let i = messages.length - 1; i >= 0; i -= 1) { + const message = messages[i]; + if (message?.role !== "user") continue; + const text = blockText(message.content); + if (text) return text; + } + return ""; +} + +/** Stable segment id for a tool result: `tool-` when known, else positional. */ +export function canonicalToolResultSegmentId(messageIndex: number, blockIndex: number, toolCallId?: string): string { + return toolCallId ? `tool-${toolCallId}` : `message-${messageIndex}-block-${blockIndex}`; +} + +export function buildCanonicalReductionTurnContext( + envelope: HostRequestEnvelope, + options?: { disclosedReadPaths?: string[] }, +): { + turnCtx: RuntimeTurnContext; + bindings: SegmentBinding[]; + diagnostics: CanonicalReductionSummary["diagnostics"]; +} { + const segments: ContextSegment[] = []; + const bindings: SegmentBinding[] = []; + const query = latestUserQuery(envelope.messages); + const toolCallHints = new Map(); + let toolLikeMessages = 0; + + envelope.messages.forEach((message) => { + if (!Array.isArray(message.content)) return; + for (const block of message.content) { + if (block.type !== "tool_call" || !block.toolCallId) continue; + toolCallHints.set(block.toolCallId, { + toolName: block.toolName, + path: extractPathHint(block.argumentsJson), + }); + } + }); + + envelope.messages.forEach((message, messageIndex) => { + if (!Array.isArray(message.content) || message.metadata?.lightrsiPassThrough === true) return; + let countedToolLike = false; + message.content.forEach((block: RuntimeContentBlock, blockIndex) => { + if (block.type !== "tool_result") return; + if (!countedToolLike) { + toolLikeMessages += 1; + countedToolLike = true; + } + if (!block.text) return; + const hint = block.toolCallId ? toolCallHints.get(block.toolCallId) : undefined; + const toolName = block.toolName ?? hint?.toolName; + const id = canonicalToolResultSegmentId(messageIndex, blockIndex, block.toolCallId); + segments.push(segmentForText({ id, text: block.text, toolName, latestUserQuery: query, path: hint?.path })); + bindings.push({ segmentId: id, messageIndex, blockIndex, toolName }); + }); + }); + + return { + turnCtx: { + sessionId: envelope.session.sessionId, + sessionMode: "single", + provider: envelope.session.host.hostId, + model: envelope.model, + apiFamily: "openai-completions", + prompt: query, + budget: { maxInputTokens: 0, reserveOutputTokens: 0 }, + segments, + metadata: { + latestUserQuery: query, + ...(options?.disclosedReadPaths ? { disclosedReadPaths: options.disclosedReadPaths } : {}), + }, + }, + bindings, + diagnostics: { + messageCount: envelope.messages.length, + toolLikeMessages, + candidateSegments: segments.length, + candidateChars: segments.reduce((sum, segment) => sum + segment.text.length, 0), + }, + }; +} + +function analyzerInstructions(segments: ContextSegment[], config: CanonicalAdapterConfig): ReductionInstruction[] { + const instructions: ReductionInstruction[] = []; + if (config.reduction.passes.toolPayloadTrim) { + instructions.push(...analyzeToolPayloadTrim(segments, { + enabled: true, + minChars: 120, + minSavedChars: Math.max(300, Math.floor(config.reduction.maxToolChars * 0.25)), + onlyLikelyToolSegments: false, + }).instructions); + } + if (config.reduction.passes.execOutputTruncation) { + const toolThresholds = config.reduction.passOptions.execOutputTruncation?.toolThresholds; + instructions.push(...analyzeExecOutputTruncation(segments, { + enabled: true, + toolThresholds: toolThresholds && typeof toolThresholds === "object" + ? toolThresholds as Record + : {}, + minExcessChars: 1000, + }).instructions); + } + return instructions; +} + +function fallbackInstructions(segments: ContextSegment[], config: CanonicalAdapterConfig): ReductionInstruction[] { + const instructions: ReductionInstruction[] = []; + const byKind = new Map(); + const execSegmentIds: string[] = []; + + for (const segment of segments) { + const meta = segment.metadata ?? {}; + const toolPayload = asRecord(meta.toolPayload); + const toolName = typeof toolPayload.toolName === "string" && toolPayload.toolName.trim() + ? toolPayload.toolName.trim() + : "tool"; + const payloadKind = typeof meta.payloadKind === "string" ? meta.payloadKind : payloadKindForText(segment.text); + const isWeb = looksLikeWebPayload(segment.text); + + if (config.reduction.passes.toolPayloadTrim && segment.text.length >= (isWeb ? 120 : 200)) { + const key = payloadKind === "json" || payloadKind === "stderr" || payloadKind === "blob" ? payloadKind : "stdout"; + const existing = byKind.get(key) ?? []; + existing.push(segment); + byKind.set(key, existing); + } + const execThreshold = thresholdForTool(toolName, config); + if (config.reduction.passes.execOutputTruncation && Number.isFinite(execThreshold) && segment.text.length > execThreshold) { + execSegmentIds.push(segment.id); + } + } + + for (const [kind, kindSegments] of byKind) { + const totalChars = kindSegments.reduce((sum, segment) => sum + segment.text.length, 0); + const hasWeb = kindSegments.some((segment) => looksLikeWebPayload(segment.text)); + const minGroupChars = hasWeb + ? Math.max(300, Math.floor(config.reduction.maxToolChars * 0.25)) + : Math.max(600, Math.floor(config.reduction.maxToolChars * 0.5)); + if (totalChars < minGroupChars) continue; + instructions.push({ + strategy: "tool_payload_trim", + segmentIds: kindSegments.map((segment) => segment.id), + confidence: kind === "json" ? 0.9 : kind === "stdout" ? 0.8 : 0.75, + priority: hasWeb ? 9 : 8, + rationale: `canonical tool payload trim: ${kindSegments.length} ${kind} segment(s), ${totalChars} chars`, + parameters: { payloadKind: kind, segmentCount: kindSegments.length, totalChars }, + }); + } + if (execSegmentIds.length > 0) { + instructions.push({ + strategy: "exec_output_truncation", + segmentIds: execSegmentIds, + confidence: 0.99, + priority: 10, + rationale: `canonical large tool output truncation: ${execSegmentIds.length} segment(s)`, + }); + } + return instructions.sort((a, b) => b.priority - a.priority); +} + +function dedupeInstructions(instructions: ReductionInstruction[]): ReductionInstruction[] { + const seen = new Set(); + const result: ReductionInstruction[] = []; + for (const instruction of instructions) { + const key = `${instruction.strategy}:${[...instruction.segmentIds].sort().join(",")}:${JSON.stringify(instruction.parameters ?? {})}`; + if (seen.has(key)) continue; + seen.add(key); + result.push(instruction); + } + return result.sort((a, b) => (b.priority ?? 0) - (a.priority ?? 0)); +} + +function withReductionPolicy(turnCtx: RuntimeTurnContext, instructions: ReductionInstruction[]): RuntimeTurnContext { + if (instructions.length === 0) return turnCtx; + const policy = asRecord(turnCtx.metadata?.policy); + const decisions = asRecord(policy.decisions); + const reduction = asRecord(decisions.reduction); + const existing = Array.isArray(reduction.instructions) ? reduction.instructions as unknown[] : []; + return { + ...turnCtx, + metadata: { + ...(turnCtx.metadata ?? {}), + policy: { + ...policy, + decisions: { + ...decisions, + reduction: { ...reduction, instructions: [...existing, ...instructions] }, + }, + }, + }, + }; +} + +export function canonicalPassOptions(config: CanonicalAdapterConfig): Record> { + return { + read_state_compaction: config.reduction.passOptions.readStateCompaction ?? {}, + tool_payload_trim: { + maxChars: config.reduction.maxToolChars, + ...(config.reduction.passOptions.toolPayloadTrim ?? {}), + }, + html_slimming: config.reduction.passOptions.htmlSlimming ?? {}, + exec_output_truncation: config.reduction.passOptions.execOutputTruncation ?? {}, + agents_startup_optimization: config.reduction.passOptions.agentsStartupOptimization ?? {}, + }; +} + +export function canonicalEnabledPassIds(config: CanonicalAdapterConfig): Set { + const ids = new Set(); + if (config.reduction.passes.readStateCompaction) ids.add("read_state_compaction"); + if (config.reduction.passes.toolPayloadTrim) ids.add("tool_payload_trim"); + if (config.reduction.passes.htmlSlimming) ids.add("html_slimming"); + if (config.reduction.passes.execOutputTruncation) ids.add("exec_output_truncation"); + if (config.reduction.passes.agentsStartupOptimization) ids.add("agents_startup_optimization"); + return ids; +} + +/** + * Build the exact inputs the shared pipeline receives (turn context with policy + * instructions and the enabled before-call passes). Exported so tests can call + * `runReductionBeforeCall` directly and assert that the adapter adds no drift. + */ +export function prepareCanonicalReductionInputs( + envelope: HostRequestEnvelope, + config: CanonicalAdapterConfig, + options?: { disclosedReadPaths?: string[] }, +) { + const built = buildCanonicalReductionTurnContext(envelope, options); + const instructions = dedupeInstructions([ + ...analyzerInstructions(built.turnCtx.segments, config), + ...fallbackInstructions(built.turnCtx.segments, config), + ]); + const turnCtx = withReductionPolicy(built.turnCtx, instructions); + const enabled = canonicalEnabledPassIds(config); + const passes = resolveReductionPasses({ + maxToolChars: config.reduction.maxToolChars, + passOptions: canonicalPassOptions(config), + }).filter((pass) => pass.phase === "before_call" && enabled.has(pass.id)); + return { turnCtx, passes, bindings: built.bindings, diagnostics: built.diagnostics }; +} + +function emptySummary( + diagnostics: CanonicalReductionSummary["diagnostics"], + skippedReason: string, + chars = 0, +): CanonicalReductionSummary { + return { + changedMessages: 0, + changedBlocks: 0, + savedChars: 0, + beforeChars: chars, + afterChars: chars, + report: [], + passEffects: [], + diagnostics, + skippedReason, + }; +} + +/** + * Reduce tool-result text in a canonical envelope. Returns a new envelope (the + * input is never mutated) and a summary shaped like Claude Code's. + */ +export async function reduceCanonicalEnvelope(params: { + envelope: HostRequestEnvelope; + config: CanonicalAdapterConfig; + memo?: ReductionMemo; +}): Promise<{ envelope: HostRequestEnvelope; summary: CanonicalReductionSummary }> { + const { envelope, config, memo } = params; + const noDiagnostics = { messageCount: envelope.messages.length, toolLikeMessages: 0, candidateSegments: 0, candidateChars: 0 }; + if (!config.modules.reduction) { + return { envelope, summary: emptySummary(noDiagnostics, "disabled") }; + } + + const presentSegmentIds = new Set(buildCanonicalReductionTurnContext(envelope).bindings.map((binding) => binding.segmentId)); + const prepared = prepareCanonicalReductionInputs(envelope, config, { + disclosedReadPaths: memo?.carriedDisclosedReadPaths(presentSegmentIds), + }); + const totalChars = prepared.diagnostics.candidateChars; + if (prepared.turnCtx.segments.length === 0 || totalChars < config.reduction.triggerMinChars) { + return { + envelope, + summary: emptySummary( + prepared.diagnostics, + prepared.turnCtx.segments.length === 0 ? "no_candidate_segments" : "below_trigger_min_chars", + totalChars, + ), + }; + } + + const { turnCtx: reducedCtx, report } = await runReductionBeforeCall({ + turnCtx: prepared.turnCtx, + passes: prepared.passes, + }); + const passEffects = report.map((entry) => ({ + id: String(entry.id), + changed: entry.changed, + skippedReason: entry.skippedReason, + beforeChars: entry.beforeChars, + afterChars: entry.afterChars, + savedChars: Math.max(0, entry.beforeChars - entry.afterChars), + touchedSegmentIds: entry.touchedSegmentIds, + })); + const changedSegmentIds = new Set(); + const trimmedSegmentIds = new Set(); + for (const entry of report) { + if (!entry.changed) continue; + for (const id of entry.touchedSegmentIds ?? []) { + changedSegmentIds.add(id); + if (entry.id === "tool_payload_trim") trimmedSegmentIds.add(id); + } + } + // Only a `read`/`file_read` result actually trimmed in this request can own a + // disclosed path (not, say, a write to the same file or an untrimmed read). + const eligibleOwnerIds = new Set(prepared.bindings + .filter((binding) => trimmedSegmentIds.has(binding.segmentId) && isReadTool(binding.toolName)) + .map((binding) => binding.segmentId)); + memo?.recordDisclosedReadPaths(reducedCtx.metadata?.disclosedReadPaths, prepared.turnCtx.segments, eligibleOwnerIds); + if (changedSegmentIds.size === 0) { + return { + envelope, + summary: { ...emptySummary(prepared.diagnostics, "pipeline_no_effect", totalChars), report, passEffects }, + }; + } + + const segmentMap = new Map(reducedCtx.segments.map((segment) => [segment.id, segment])); + const messages = envelope.messages.slice(); + const changedMessages = new Set(); + const visualSegments: CanonicalReductionVisualSegment[] = []; + let changedBlocks = 0; + let savedChars = 0; + + for (const binding of prepared.bindings) { + if (!changedSegmentIds.has(binding.segmentId)) continue; + const segment = segmentMap.get(binding.segmentId); + const message = messages[binding.messageIndex]; + if (!segment || !message || !Array.isArray(message.content)) continue; + const block = message.content[binding.blockIndex]; + if (!block || block.type !== "tool_result" || block.text === segment.text) continue; + + const nextText = segment.text; + const before = block.text; + const content = message.content.slice(); + content[binding.blockIndex] = { ...block, text: nextText }; + messages[binding.messageIndex] = { ...message, content }; + + changedBlocks += 1; + changedMessages.add(binding.messageIndex); + const segmentSaved = Math.max(0, before.length - nextText.length); + savedChars += segmentSaved; + visualSegments.push({ + segmentId: binding.segmentId, + messageIndex: binding.messageIndex, + blockIndex: binding.blockIndex, + toolName: binding.toolName, + savedChars: segmentSaved, + beforeText: before, + afterText: nextText, + report: report.filter((entry) => entry.changed && entry.touchedSegmentIds?.includes(binding.segmentId)), + }); + } + + return { + envelope: changedBlocks > 0 ? { ...envelope, messages } : envelope, + summary: { + changedMessages: changedMessages.size, + changedBlocks, + savedChars, + beforeChars: totalChars, + afterChars: Math.max(0, totalChars - savedChars), + report, + passEffects, + diagnostics: prepared.diagnostics, + visualSegments, + }, + }; +} + +/** Characters removed per reduction pass in one request (only passes that changed text). */ +export function passSavedChars(summary: CanonicalReductionSummary | undefined): Record { + const out: Record = {}; + for (const effect of summary?.passEffects ?? []) { + if (effect.changed && effect.savedChars > 0) out[effect.id] = (out[effect.id] ?? 0) + effect.savedChars; + } + return out; +} diff --git a/components/adapters/shared/cli-context.ts b/components/adapters/shared/cli-context.ts index aafd1f8b..282331a8 100644 --- a/components/adapters/shared/cli-context.ts +++ b/components/adapters/shared/cli-context.ts @@ -1,7 +1,7 @@ import { mkdir, readFile, writeFile } from "node:fs/promises"; import { dirname, join } from "node:path"; -type SharedCliHostId = "codex" | "claude-code"; +type SharedCliHostId = "codex" | "claude-code" | "opencode"; type SharedCliHostPathOverrides = { tokenPilotConfigPath?: string; diff --git a/components/products/cli/build.ts b/components/products/cli/build.ts index 584b3150..698c8691 100644 --- a/components/products/cli/build.ts +++ b/components/products/cli/build.ts @@ -1,9 +1,13 @@ import { build } from "esbuild"; +import { join } from "node:path"; async function main() { await build({ entryPoints: ["src/cli.ts"], bundle: true, + // adapters/shared/canonical (pulled in by the OpenCode host) has no node_modules + // of its own; resolve its workspace dependencies through an adapter that declares them. + nodePaths: [join(__dirname, "node_modules"), join(__dirname, "..", "..", "adapters", "opencode", "node_modules")], outfile: "dist/cli.js", platform: "node", target: "node20", diff --git a/components/products/cli/src/dispatch.ts b/components/products/cli/src/dispatch.ts index 5b05f4f5..1b46df95 100644 --- a/components/products/cli/src/dispatch.ts +++ b/components/products/cli/src/dispatch.ts @@ -42,6 +42,12 @@ function currentEnvPathOverrides(host: CliHostId): CliHostPathOverrides | undefi ? { tokenPilotConfigPath, hostConfigPath, hostAuxConfigPath } : undefined; } + if (host === "opencode") { + const tokenPilotConfigPath = process.env.TOKENPILOT_OPENCODE_CONFIG?.trim(); + const configDir = process.env.TOKENPILOT_OPENCODE_CONFIG_DIR?.trim(); + const hostConfigPath = configDir ? `${configDir.replace(/[\\/]+$/, "")}/opencode.json` : undefined; + return tokenPilotConfigPath || hostConfigPath ? { tokenPilotConfigPath, hostConfigPath } : undefined; + } return undefined; } diff --git a/components/products/cli/src/hosts/factory.ts b/components/products/cli/src/hosts/factory.ts index 518fc3f4..a083f95b 100644 --- a/components/products/cli/src/hosts/factory.ts +++ b/components/products/cli/src/hosts/factory.ts @@ -5,6 +5,7 @@ import { registerCleanCommandBackendResolver } from "../clean.js"; import { createClaudeCodeCleanCommandBackend, createClaudeCodeCliBridge } from "./claude-code.js"; import { createCodexCleanCommandBackend, createCodexCliBridge } from "./codex.js"; import { createOpenClawCleanCommandBackend, createOpenClawCliBridge } from "./openclaw.js"; +import { createOpenCodeCliBridge } from "./opencode.js"; import { CLI_HOSTS, getCliHostRegistration, @@ -34,6 +35,9 @@ const CLI_HOST_REGISTRATIONS: CliHostRegistration[] = CLI_HOSTS.map((host) => ({ pathOverrides: target.pathOverrides, }); } + if (host.hostId === "opencode") { + return createOpenCodeCliBridge({ host: "opencode", sessionId: target.sessionId, pathOverrides: target.pathOverrides }); + } const bridge = createOpenClawCliBridge({ host: "openclaw", sessionId: target.sessionId }); const handler = createProductSurfaceCommandHandler({ bridge: bridge.bridge, diff --git a/components/products/cli/src/hosts/in-process.ts b/components/products/cli/src/hosts/in-process.ts new file mode 100644 index 00000000..3e2737fd --- /dev/null +++ b/components/products/cli/src/hosts/in-process.ts @@ -0,0 +1,146 @@ +/** + * Shared CLI bridge for in-process canonical adapters (currently OpenCode). + * + * Mirrors the Claude Code / Codex bridges: flat `tokenpilot.json`, the restricted + * host command handler (status, report, doctor, visual, mode conservative|normal, + * stabilizer, reduction, eviction), and the shared report/visual surfaces. Only the + * per-host pieces (config paths, doctor, display name) are supplied by the host. + */ +import { mkdir } from "node:fs/promises"; +import { dirname } from "node:path"; +import { + ProductSurfaceConfigAdapter, + ProductSurfaceHostBridge, + readLatestUxEffect, + readUxSessionAggregate, + resolveLatestSessionId, +} from "@lightrsi/host-adapter"; +import { formatDisplayValue, formatOnOff, getNestedValue } from "@lightrsi/product-surface"; +import type { CliHostPathOverrides } from "../context-store.js"; +import { + applyStandardRuntimeModeConfig, + buildSessionReportResult, + createRestrictedHostCommandHandler, + resolveConfiguredPreferredSessionId, + resolvePreferredSessionId, +} from "./shared.js"; +import { handleStandaloneVisualCommandWithSelection } from "./visual.js"; + +export const IN_PROCESS_REDUCTION_PASS_NAMES = [ + "readStateCompaction", + "toolPayloadTrim", + "htmlSlimming", + "execOutputTruncation", + "agentsStartupOptimization", +] as const; + +export type InProcessHostSpec = { + hostId: "opencode"; + displayName: string; + defaultConfigPath(): string; + loadConfig(configPath: string): Promise>; + normalizeConfig(raw: Record, configPath: string): Record; + writeConfig(config: Record, configPath: string): Promise; + configAdapter: ProductSurfaceConfigAdapter; + resolveStateDir(config: Record): string | undefined; + doctor(params: { config: Record; configPath: string; hostConfigPath?: string }): Promise; + /** Extra status lines after the shared ones. */ + statusLines?(config: Record): string[]; +}; + +export function formatInProcessStatus(spec: InProcessHostSpec, config: Record): string { + const evictionOn = Boolean(getNestedValue(config, ["modules", "eviction"])) && Boolean(getNestedValue(config, ["eviction", "enabled"])); + return [ + `TokenPilot ${spec.displayName} status:`, + `- enabled: ${formatOnOff(config.enabled)}`, + `- stabilizer: ${formatOnOff(getNestedValue(config, ["modules", "stabilizer"]))}`, + `- dynamicContextTarget: ${formatDisplayValue(getNestedValue(config, ["hooks", "dynamicContextTarget"]))}`, + `- reduction: ${formatOnOff(getNestedValue(config, ["modules", "reduction"]))}`, + `- triggerMinChars: ${formatDisplayValue(getNestedValue(config, ["reduction", "triggerMinChars"]))}`, + `- maxToolChars: ${formatDisplayValue(getNestedValue(config, ["reduction", "maxToolChars"]))}`, + `- eviction: ${formatOnOff(evictionOn)}`, + `- evictionMinBlockChars: ${formatDisplayValue(getNestedValue(config, ["eviction", "minBlockChars"]))}`, + `- stateDir: ${formatDisplayValue(config.stateDir)}`, + ...(spec.statusLines?.(config) ?? []), + ].join("\n"); +} + +export function createInProcessCliBridge(spec: InProcessHostSpec, target: { + sessionId?: string; + pathOverrides?: CliHostPathOverrides; +}): { + bridge: ProductSurfaceHostBridge; + configAdapter: ProductSurfaceConfigAdapter; + maybeResolveLatestSessionId(): Promise; + resolveSessionId(sessionId?: string): Promise; + handleCommand(ctx: { args: string; sessionId?: string }): Promise<{ text: string }>; +} { + const configPath = target.pathOverrides?.tokenPilotConfigPath?.trim() || spec.defaultConfigPath(); + const hostConfigPath = target.pathOverrides?.hostConfigPath?.trim() || undefined; + const loadConfig = () => spec.loadConfig(configPath); + const writeConfig = async (next: Record) => { + await mkdir(dirname(configPath), { recursive: true }); + await spec.writeConfig(spec.normalizeConfig(next, configPath), configPath); + }; + + const bridge: ProductSurfaceHostBridge = { + loadConfig, + writeConfig, + async handleDoctor(currentConfig) { + return { text: await spec.doctor({ config: currentConfig, configPath, hostConfigPath }) }; + }, + async handleVisual(currentConfig) { + const stateDir = spec.resolveStateDir(currentConfig); + if (!stateDir) return { text: "TokenPilot stateDir is not configured." }; + const sessionId = await resolvePreferredSessionId({ + explicitSessionId: target.sessionId, + stateDir, + resolveLatestSessionId, + readLatestUxEffect, + }); + return handleStandaloneVisualCommandWithSelection({ host: spec.hostId, sessionId }); + }, + async handleReport(_ctx, currentConfig) { + return buildSessionReportResult({ + currentConfig, + explicitSessionId: target.sessionId, + configAdapter: spec.configAdapter, + resolveLatestSessionId, + readLatestUxEffect, + readSessionAggregate: readUxSessionAggregate, + }); + }, + }; + + const handleCommand = createRestrictedHostCommandHandler({ + displayName: spec.displayName, + cliHostName: spec.hostId, + reductionPassNames: IN_PROCESS_REDUCTION_PASS_NAMES, + supportsEviction: true, + bridge, + configAdapter: spec.configAdapter, + loadConfig, + formatStatus: (config) => formatInProcessStatus(spec, config), + async applyMode(mode) { + await writeConfig(applyStandardRuntimeModeConfig(await loadConfig(), mode)); + }, + }); + + return { + bridge, + configAdapter: spec.configAdapter, + maybeResolveLatestSessionId() { + return resolveConfiguredPreferredSessionId({ + loadConfig, + resolveStateDir: spec.resolveStateDir, + resolveLatestSessionId, + readLatestUxEffect, + }); + }, + async resolveSessionId(sessionId?: string) { + const text = typeof sessionId === "string" ? sessionId.trim() : ""; + return text || undefined; + }, + handleCommand, + }; +} diff --git a/components/products/cli/src/hosts/opencode.ts b/components/products/cli/src/hosts/opencode.ts new file mode 100644 index 00000000..01b45925 --- /dev/null +++ b/components/products/cli/src/hosts/opencode.ts @@ -0,0 +1,38 @@ +import { dirname } from "node:path"; +import { + defaultTokenPilotOpenCodeConfigPath, + loadTokenPilotOpenCodeConfig, + normalizeTokenPilotOpenCodeConfig, + writeTokenPilotOpenCodeConfig, + type TokenPilotOpenCodeConfig, +} from "../../../../adapters/opencode/src/config.js"; +import { formatOpenCodeDoctorReport, inspectOpenCodeDoctor } from "../../../../adapters/opencode/src/doctor.js"; +import { + openCodeProductSurfaceConfigAdapter, + resolveOpenCodeStateDir, +} from "../../../../adapters/opencode/src/host-config-adapter.js"; +import type { CliHostPathOverrides } from "../context-store.js"; +import { createInProcessCliBridge, type InProcessHostSpec } from "./in-process.js"; + +export const OPENCODE_CLI_HOST_SPEC: InProcessHostSpec = { + hostId: "opencode", + displayName: "OpenCode", + defaultConfigPath: defaultTokenPilotOpenCodeConfigPath, + loadConfig: async (configPath) => (await loadTokenPilotOpenCodeConfig(configPath)) as unknown as Record, + normalizeConfig: (raw, configPath) => normalizeTokenPilotOpenCodeConfig(raw, { configPath }) as unknown as Record, + writeConfig: (config, configPath) => writeTokenPilotOpenCodeConfig(config as unknown as TokenPilotOpenCodeConfig, configPath), + configAdapter: openCodeProductSurfaceConfigAdapter, + resolveStateDir: resolveOpenCodeStateDir, + async doctor({ config, configPath, hostConfigPath }) { + const report = await inspectOpenCodeDoctor({ + config: config as unknown as TokenPilotOpenCodeConfig, + configPath, + ...(hostConfigPath ? { configDir: dirname(hostConfigPath) } : {}), + }); + return formatOpenCodeDoctorReport(report); + }, +}; + +export function createOpenCodeCliBridge(target: { host: "opencode"; sessionId?: string; pathOverrides?: CliHostPathOverrides }) { + return createInProcessCliBridge(OPENCODE_CLI_HOST_SPEC, target); +} diff --git a/components/products/cli/src/hosts/registry.ts b/components/products/cli/src/hosts/registry.ts index 05c11bac..74ffa83a 100644 --- a/components/products/cli/src/hosts/registry.ts +++ b/components/products/cli/src/hosts/registry.ts @@ -3,6 +3,7 @@ import { ProductHostRegistry, type ProductHostRegistration } from "@lightrsi/pro import { CLAUDE_CODE_PRODUCT_HOST_REGISTRATION } from "../../../../adapters/claude-code/src/product-registration.js"; import { CODEX_PRODUCT_HOST_REGISTRATION } from "../../../../adapters/codex/src/product-registration.js"; import { OPENCLAW_PRODUCT_HOST_REGISTRATION } from "../../../../adapters/openclaw/src/product-registration.js"; +import { OPENCODE_PRODUCT_HOST_REGISTRATION } from "../../../../adapters/opencode/src/product-registration.js"; import { readCliHostPathOverrides, type CliHostPathOverrides } from "../context-store.js"; export type CliHostRuntime = { @@ -31,6 +32,7 @@ export const CLI_HOSTS = [ OPENCLAW_PRODUCT_HOST_REGISTRATION, CODEX_PRODUCT_HOST_REGISTRATION, CLAUDE_CODE_PRODUCT_HOST_REGISTRATION, + OPENCODE_PRODUCT_HOST_REGISTRATION, ] as const; export type CliHostId = (typeof CLI_HOSTS)[number]["hostId"]; @@ -54,7 +56,9 @@ async function productConfigPath(hostId: CliHostId): Promise ? process.env.TOKENPILOT_CODEX_CONFIG?.trim() : hostId === "claude-code" ? process.env.TOKENPILOT_CLAUDE_CODE_CONFIG?.trim() - : undefined; + : hostId === "opencode" + ? process.env.TOKENPILOT_OPENCODE_CONFIG?.trim() + : undefined; return environmentPath || (await readCliHostPathOverrides(hostId))?.tokenPilotConfigPath?.trim(); } diff --git a/components/products/cli/src/usage.ts b/components/products/cli/src/usage.ts index f3004166..747a4f7e 100644 --- a/components/products/cli/src/usage.ts +++ b/components/products/cli/src/usage.ts @@ -34,6 +34,7 @@ export function formatCliUsage(): string { " lightrsi openclaw doctor", " lightrsi openclaw clean --require-tty --session ", " lightrsi claude-code doctor", + " lightrsi opencode report", " lightrsi codex session clean", " lightrsi openclaw session 123e4567-e89b-12d3-a456-426614174000 report", " lightrsi use openclaw", diff --git a/components/products/cli/tests/in-process-hosts.test.ts b/components/products/cli/tests/in-process-hosts.test.ts new file mode 100644 index 00000000..d08b36db --- /dev/null +++ b/components/products/cli/tests/in-process-hosts.test.ts @@ -0,0 +1,137 @@ +/** + * Regression matrix: OpenCode registration in the shared CLI + * (hosts/in-process.ts, hosts/opencode.ts, registry, factory, dispatch, usage) + * + * Registration + * H1 CLI_HOSTS lists opencode after the existing hosts (order preserved) + * H2 parseCliHostId accepts "opencode" + * H3 usage lists the host and an example + * Command surface + * C1 status prints every shared key for the host's config + * C2 mode conservative / normal write the standard presets; aggressive refused; + * anything else prints usage + * C3 reduction pass is written; unknown pass names are rejected + * C4 eviction on sets modules.eviction, eviction.enabled, taskStateEstimator.enabled; + * eviction set minBlockChars writes eviction.minBlockChars; other subcommands refused + * C5 stabilizer target user writes hooks.dynamicContextTarget + * C6 doctor returns the host's own doctor report + * C7 report: no activity → "No TokenPilot session stats yet."; with activity → + * report for the latest session + * C8 the TOKENPILOT_OPENCODE_CONFIG env override selects the config file + */ +import assert from "node:assert/strict"; +import { after, describe, it } from "node:test"; +import { mkdtemp, readFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { recordUxEffect, writeLatestSessionRef } from "@lightrsi/host-adapter"; + +import { dispatchCli } from "../src/dispatch.js"; +import { CLI_HOSTS, parseCliHostId } from "../src/hosts/registry.js"; +import { formatCliUsage } from "../src/usage.js"; + +const saved = { + HOME: process.env.HOME, + USERPROFILE: process.env.USERPROFILE, + TOKENPILOT_OPENCODE_CONFIG: process.env.TOKENPILOT_OPENCODE_CONFIG, + TOKENPILOT_OPENCODE_CONFIG_DIR: process.env.TOKENPILOT_OPENCODE_CONFIG_DIR, +}; +after(() => { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } +}); + +const HOSTS = [ + { id: "opencode", env: "TOKENPILOT_OPENCODE_CONFIG", doctorTitle: "TokenPilot OpenCode doctor:", statusTitle: "TokenPilot OpenCode status:" }, +] as const; + +async function isolate(host: (typeof HOSTS)[number]): Promise<{ configPath: string; home: string }> { + const home = await mkdtemp(join(tmpdir(), `lightrsi-cli-${host.id}-`)); + process.env.HOME = home; + process.env.USERPROFILE = home; + const configPath = join(home, host.id, "tokenpilot.json"); + process.env[host.env] = configPath; + if (host.id === "opencode") process.env.TOKENPILOT_OPENCODE_CONFIG_DIR = join(home, "opencode"); + return { configPath, home }; +} + +const readConfig = async (path: string) => JSON.parse(await readFile(path, "utf8")); + +describe("registration", () => { + it("H1 hosts are appended in order", () => { + assert.deepEqual(CLI_HOSTS.map((h) => h.hostId), ["openclaw", "codex", "claude-code", "opencode"]); + }); + it("H2 host ids parse", () => { + assert.equal(parseCliHostId("opencode"), "opencode"); + }); + it("H3 usage lists the host", () => { + const usage = formatCliUsage(); + for (const text of [" opencode\n", "lightrsi opencode report"]) assert.ok(usage.includes(text), text); + }); +}); + +for (const host of HOSTS) { + describe(`${host.id} command surface`, () => { + it("C1 status lists every shared key", async () => { + await isolate(host); + const { text } = await dispatchCli([host.id, "status"]); + assert.ok(text.startsWith(host.statusTitle)); + for (const key of ["enabled", "stabilizer", "dynamicContextTarget", "reduction", "triggerMinChars", "maxToolChars", "eviction", "evictionMinBlockChars", "stateDir"]) { + assert.ok(text.includes(`- ${key}:`), key); + } + }); + it("C2 modes", async () => { + const { configPath } = await isolate(host); + assert.equal((await dispatchCli([host.id, "mode", "conservative"])).text, "✅ Runtime mode = conservative"); + assert.deepEqual([(await readConfig(configPath)).reduction.triggerMinChars, (await readConfig(configPath)).reduction.maxToolChars], [4000, 1800]); + await dispatchCli([host.id, "mode", "normal"]); + assert.deepEqual([(await readConfig(configPath)).reduction.triggerMinChars, (await readConfig(configPath)).reduction.maxToolChars], [2200, 1200]); + assert.match((await dispatchCli([host.id, "mode", "aggressive"])).text, /does not support lifecycle eviction mode/); + assert.match((await dispatchCli([host.id, "mode", "turbo"])).text, /^Usage: lightrsi .* mode /); + }); + it("C3 reduction passes", async () => { + const { configPath } = await isolate(host); + await dispatchCli([host.id, "reduction", "pass", "htmlSlimming", "off"]); + assert.equal((await readConfig(configPath)).reduction.passes.htmlSlimming, false); + assert.match((await dispatchCli([host.id, "reduction", "pass", "bogus", "off"])).text, /supports only these passes/); + }); + it("C4 eviction controls", async () => { + const { configPath } = await isolate(host); + await dispatchCli([host.id, "eviction", "on"]); + const on = await readConfig(configPath); + assert.deepEqual([on.modules.eviction, on.eviction.enabled, on.taskStateEstimator.enabled], [true, true, true]); + await dispatchCli([host.id, "eviction", "set", "minBlockChars", "8000"]); + assert.equal((await readConfig(configPath)).eviction.minBlockChars, 8000); + assert.match((await dispatchCli([host.id, "eviction", "policy", "lru"])).text, /eviction supports only status, on\|off, and set minBlockChars/); + }); + it("C5 stabilizer target", async () => { + const { configPath } = await isolate(host); + await dispatchCli([host.id, "stabilizer", "target", "user"]); + assert.equal((await readConfig(configPath)).hooks.dynamicContextTarget, "user"); + }); + it("C6 doctor is the host's report", async () => { + await isolate(host); + const { text } = await dispatchCli([host.id, "doctor"]); + assert.ok(text.startsWith(host.doctorTitle), text.split("\n")[0]); + }); + it("C7 report before and after activity", async () => { + const { configPath } = await isolate(host); + await dispatchCli([host.id, "mode", "normal"]); + assert.equal((await dispatchCli([host.id, "report"])).text, "No TokenPilot session stats yet."); + const stateDir = (await readConfig(configPath)).stateDir as string; + await writeLatestSessionRef(stateDir, "sess-report", new Date().toISOString()); + await recordUxEffect(stateDir, { at: new Date().toISOString(), sessionId: "sess-report", model: "qwen3", countMode: "chars", beforeCount: 1000, afterCount: 400, savedCount: 600 }); + const { text } = await dispatchCli([host.id, "report"]); + assert.match(text, /TokenPilot report:/); + assert.match(text, /sess-report/); + }); + it("C8 the env override selects the config file", async () => { + const { configPath } = await isolate(host); + await dispatchCli([host.id, "mode", "conservative"]); + assert.equal((await readConfig(configPath)).reduction.maxToolChars, 1800); + }); + }); +} diff --git a/docs/adapters/opencode-design.md b/docs/adapters/opencode-design.md new file mode 100644 index 00000000..c262f4f0 --- /dev/null +++ b/docs/adapters/opencode-design.md @@ -0,0 +1,164 @@ +# TokenPilot adapter design: OpenCode + +Status: implemented in `components/adapters/opencode/`. This note was written before implementation and updated afterwards to match what was built. + +## Pinned versions + +| Item | Value | +| :-- | :-- | +| LightRSI base commit | `3b7f28d76c98de85999cf94ae99d39ad144fa327` | +| OpenCode | `opencode-ai@1.18.33` (npm `latest` on 2026-09-28); source tag `v1.18.33` of `anomalyco/opencode` (commit `51ef4be`) | +| Plugin API package | `@opencode-ai/plugin@1.18.33`, types from `@opencode-ai/sdk@1.18.33` | + +Every "verified" statement cites a path in that source tag (`packages/opencode/src/...`, +`packages/core/src/...`) or in the published plugin package (`dist/index.d.ts`). + +## Plugin API choice: v1 `Hooks` + +`@opencode-ai/plugin@1.18.33` ships both the v1 `Hooks` interface (`dist/index.d.ts`) and +a v2 API (`dist/v2/promise`, `Plugin.define`). The v2 `PluginContext` in this version +exposes only the `agent`, `aisdk`, `catalog`, `command`, `integration`, `plugin`, +`reference` and `skill` domains (`dist/v2/promise/context.d.ts`). It has **no** +message, tool or session hook, and there is no `ctx.session.hook("context")` in 1.18.33. +Every surface TokenPilot needs exists only in v1, so the adapter is a v1 plugin: +`export default { id: "tokenpilot", server }` (read by `readV1Plugin`, +`plugin/shared.ts:272`). + +## Design First (playbook questions) + +| Question | Answer for OpenCode 1.18.33 | Evidence | +| :-- | :-- | :-- | +| Can the host rewrite requests before model execution? | Yes. `experimental.chat.messages.transform` mutates the message list before each model step; `experimental.chat.system.transform` mutates the system prompt; `chat.params` mutates sampling options. | `session/prompt.ts:1255`, `session/llm/request.ts:67-72`, `:114-131` | +| Can it rewrite responses after model execution? | Partly. `tool.execute.after` can mutate a tool's `output` before it is recorded; `experimental.text.complete` can rewrite a finished text part. There is no hook for a whole assistant message. | `session/tools.ts:122` etc., plugin `dist/index.d.ts` | +| Streaming chunks, final responses, or both? | Plugins see neither raw stream nor raw response. They observe persisted parts through the `event` hook (`message.part.updated` with `delta`). | `dist/index.d.ts` `event`, SDK `EventMessagePartUpdated` | +| Transcript history exposed directly, or reconstructed? | Directly, per step: `messages.transform` receives `{ info: Message; parts: Part[] }[]`, loaded fresh from storage on every loop iteration (`MessageV2.filterCompactedEffect`, `session/prompt.ts:1092`). | `session/prompt.ts` | +| File-, hook-, plugin- or API-based? | Plugin (in-process Bun module) plus native MCP config. | `config/plugin.ts`, core `v1/config/mcp.ts` | + +## Host facts verified against 1.18.33 + +| Brief claim | Result | Evidence | +| :-- | :-- | :-- | +| `experimental.chat.system.transform` mutates the system prompt array | Confirmed. Fired per LLM request with `{ sessionID, model }` and `{ system }`. At trigger time `system` has one joined string. If a plugin appends entries, OpenCode re-joins them to `[header, rest]`. | `session/llm/request.ts:57-78`; also `agent/agent.ts:381` for agent generation | +| `tool.definition` mutates tool descriptions/schemas | Confirmed. | `tool/registry.ts:318` | +| `chat.message`, `chat.params` | Confirmed. `chat.params` output: `temperature, topP, topK, maxOutputTokens, options`. | `session/prompt.ts:1000`, `session/llm/request.ts:114` | +| `experimental.chat.messages.transform`: every model call, or only around compaction? | **Every model step.** Triggered in the main loop right before `MessageV2.toModelMessagesEffect` (`session/prompt.ts:1255`), and also for the compaction summariser's request (`session/compaction.ts:379`). Input is `{}` (no sessionID); session id comes from `messages[*].info.sessionID`. Only in-place mutation of the passed array is observed: the caller keeps using its own `msgs` binding. | `session/prompt.ts`, `session/compaction.ts` | +| `tool.execute.after` "declared but never invoked" (anomalyco/opencode #25918) | **Not reproduced in 1.18.33.** It is triggered after every built-in tool (`session/tools.ts:122`), MCP tool (`:209`, `:292`, `:374`, `:421`), task/subtask tool (`session/prompt.ts:390`) and code-mode tool (`tool/code-mode.ts:181`), with a mutable `output`. It is not used for reduction (see below). | `session/tools.ts` | +| `experimental.session.compacting` | Confirmed: `context: string[]`, optional `prompt` override. | `session/compaction.ts:374` | +| v1 vs v2 API | See "Plugin API choice" above. | | +| OpenCode prunes old tool outputs natively (last ~40k tokens protected) | Mechanism confirmed: `PRUNE_PROTECT = 40_000`, `PRUNE_MINIMUM = 20_000`, skips the last 2 user turns and the `skill` tool; pruned outputs render as `[Old tool result content cleared]`. It is **off by default in 1.18.33**: `compaction.prune` "(default: false)", and `prune()` returns early unless it is set. `OPENCODE_DISABLE_PRUNE` forces it off. | `session/compaction.ts:28-31,273-317`, core `v1/config/config.ts:154`, `config/config.ts:596`, `session/message-v2.ts:297` | +| Native MCP | Confirmed. `mcp. = { type: "local", command: string[], environment }`. The model sees tool `_`. | core `v1/config/mcp.ts`, `mcp/catalog.ts:119` | +| Hook errors | `Plugin.trigger` runs each hook through `Effect.promise` with **no catch** (`plugin/index.ts:284-297`), so a throwing hook fails the whole request. The adapter must catch everything itself to fail open. | `plugin/index.ts` | + +Other facts used: + +- Global config dir: `$XDG_CONFIG_HOME/opencode` (default `~/.config/opencode`). Files merged in order: + `config.json`, `opencode.json`, `opencode.jsonc` (`config/config.ts:272-274`). Plugins + auto-load from `{plugin,plugins}/*.{ts,js}` in that dir (`config/plugin.ts:21`). +- Native tool-output truncation happens before the model and before our hooks see the + output: default 2000 lines / 50 KiB, configurable via `tool_output.max_lines|max_bytes` + (`tool/truncate*.ts`). TokenPilot reduction therefore operates on already-truncated + outputs. +- OpenCode sends the system prompt as its own message(s) ahead of history + (`session/llm/request.ts:98-111`). Its environment block contains + `Today's date: …`, which the shared stabilizer classifies as volatile. + +## Surface table + +| Surface | OpenCode 1.18.33 | Closest existing adapter precedent | +| :-- | :-- | :-- | +| Install / uninstall flow | `install:opencode` (1) writes a marker-tagged loader `~/.config/opencode/plugins/tokenpilot.js` that re-exports the built `dist/plugin.js`; (2) registers the shared recovery MCP server under `mcp.tokenpilot_memory_fault_recover` in `~/.config/opencode/opencode.json` after writing a timestamped backup; (3) writes runtime config `~/.config/opencode/tokenpilot.json` (normal mode). If only `opencode.jsonc` exists, it is **not** rewritten (comments would be lost): the installer prints the MCP snippet and `doctor` reports it missing. `uninstall:opencode` removes the loader (marker-checked) and exactly the `mcp` key it added, and leaves every other user edit alone. | Codex / Claude Code `install.ts` (MCP registration + backups) | +| Session id, turn id, workspace root | `sessionID` from hook input (`system.transform`, tool hooks) or `messages[*].info.sessionID` (`messages.transform`); turn id = id of the last `user` message; workspace = `PluginInput.worktree` (fallback `directory`). | Claude Code hook session binding | +| Transcript bridge (host ↔ canonical) | `{info, parts}[]` ↔ `RuntimeMessage[]`. User `text` → `text`; user `file` with `image/*` mime → `image`, other files → preserved. Assistant `text` → `text`; `reasoning` → preserved verbatim (no kernel block); completed/error `tool` part → `tool_call` on the assistant message plus a following `tool` role message with a `tool_result` (`output`, `error`, or the native `[Old tool result content cleared]` placeholder when `time.compacted` is set). `step-*`, `snapshot`, `patch`, `agent`, `retry`, `compaction`, `subtask`, pending/running tools → preserved, never rewritten. Encode writes changed tool-result text into a **cloned** part and swaps the clone into the array, so storage objects are never mutated. | Claude Code `messages-codec.ts` | +| Stable-prefix path | `experimental.chat.system.transform`: run the shared `applyStablePrefixToInstructions` over `system[0]` with target `developer`, so volatile lines (`Today's date: …`) move to the tail of the system prompt. Then the shared recovery protocol is appended, idempotently, from the shared injector. **Target `user` is not supported**: in 1.18.33 `messages.transform` runs before `system.transform` in the same step (`session/prompt.ts:1255` vs `session/llm/request.ts:67`), so the volatile lines are not known when the first user message could still be changed. The config value is accepted, runs as `developer`, and is logged once and reported by `doctor`. The visual snapshot comes from the shared `buildStabilityVisualSnapshotFromEnvelopes`. | OpenClaw `root-prompt-stabilizer.ts` / shared stabilizer | +| Reduction path | `experimental.chat.messages.transform` (every model step): decode → canonical envelope → shared before-call pipeline (`prepareObservedBeforeCall` → `@lightrsi/reduction`) → encode back into cloned parts. Same request-time position as the Codex/Claude Code proxies and OpenClaw. | Claude Code `reduction.ts` + `prepareObservedBeforeCall` | +| Eviction path (opt-in) | `experimental.chat.messages.transform`. Plugins have no persistent part-edit hook, so eviction is a request-local overlay, as in Claude Code. Canonical messages → surface entries (`part:` / `msg:`) → shared `runCanonicalEvictionCycle` (`@lightrsi/eviction` canonical-surface core, lifted from DSH). Accepted replacements (stub plus recovery hint, originals archived) are stored in `/tokenpilot/eviction-overlay/.json` and re-applied identically on every later request, so decisions are durable and the prefix stays stable. It runs before reduction, like the Claude Code gateway. It is isolated, so an estimator failure never disables reduction. Off by default. | Claude Code overlay approach; DSH cycle | +| Recovery path | The shared recovery MCP server (`@lightrsi/mcp` `dist/server.js`, `TOKENPILOT_STATE_DIR` env) registered as a native OpenCode MCP server, as Codex and Claude Code do. The model sees it as `tokenpilot_memory_fault_recover_memory_fault_recover` (OpenCode's `server_tool` naming), the same way Claude Code shows `mcp__tokenpilot_memory_fault_recover__memory_fault_recover`. | Codex, Claude Code | +| Runtime state root, namespace, archive path | Config `~/.config/opencode/tokenpilot.json` (override `TOKENPILOT_OPENCODE_CONFIG`). State `~/.config/opencode/tokenpilot-state/tokenpilot/`, namespace `tokenpilot`, archives `/tokenpilot/tool-result-archives//`. | Claude Code | +| Status, report, doctor, visual | Shared CLI registration: `lightrsi opencode status|report|doctor|visual|mode|stabilizer|reduction`. `doctor` checks config, loader, built bundle, MCP registration (command, args, `TOKENPILOT_STATE_DIR`), MCP probe, `compaction.prune` setting, and last activity. | Claude Code | +| Declared feature subset (`host-binding.ts`) | `stabilizer`, `reduction`, `eviction` (eviction opt-in). | Codex, Claude Code | +| Supported modes | `conservative`, `normal` (default). `aggressive` refused. | Codex, Claude Code | + +## Behavioural decisions + +### Reduction runs at request time, not in `tool.execute.after` + +`tool.execute.after` does fire in 1.18.33, but reducing there would persist a +single-segment reduction into the session. All reference adapters reduce at request time +over the full history, which `read_state_compaction` requires. `messages.transform` is +the request-time position, so it is used. `tool.execute.after` is not hooked. + +### Byte-stable history across requests + +Reduction re-runs over the whole history on every request, so the same tool output must +produce exactly the same text each time or the provider's prefix cache misses from that +point on. The shared `artifact-store` names archives by content (#90), so a re-reduced +result keeps the same `Archive:` line on every request and across OpenCode restarts. +The adapter adds nothing for this. + +### Repeat-read detection across requests + +`tool_payload_trim` leaves a file read untrimmed when its path was already disclosed: +the model re-reading a file it has seen summarised is taken as a request for the full +body. The pass reports the disclosed paths in `disclosedReadPaths` so a caller can carry +them forward. OpenCode resends the whole history on every request, so carrying every path +makes the pass treat the *same* read as its own repeat on the next request and send it +untrimmed (the same bug fixed for Claude Code in #92). + +`ReductionMemo` therefore records which read disclosed each path, and carries a path +only once that read has left the history (compaction, eviction, native pruning). As in +#92, only a `read`/`file_read` result actually trimmed in that request can own a path, +and the record is rebuilt from the pass's bounded reported set. A disclosure whose read +is still present is re-detected by the pass from the history itself, so output is a +pure function of the history and genuine repeat reads behave as in the reference. + +The memo is saved per session to `/tokenpilot/reduction-memo/.json` +when it changes and loaded on the session's first request, so `opencode run +--continue` and TUI restarts still honour a deliberate re-read. Regression tests: +`shared-canonical-reduction` M1–M9 and D1–D5, `shared-canonical-memo-store` MS1–MS5, +`runtime` R7–R9. + +### Native pruning interaction + +TokenPilot does not change `compaction.prune`. `doctor` and `status` report its effective +value. + +- Off (the 1.18.33 default): reduction is the only thing shrinking old tool outputs, + apart from native truncation at 50 KiB / 2000 lines. +- On: once more than 40k tokens of newer tool output exist, OpenCode persistently + replaces older outputs with `[Old tool result content cleared]` (the recovery MCP + cannot restore those, but the adapter never archived them either, because they are + below the reduction thresholds). Newer outputs are still reduced request-locally by + TokenPilot. The two do not conflict: a pruned part is below `triggerMinChars` and the + passes skip it. Pruning does rewrite earlier history, so it moves the cached prefix + boundary. That is OpenCode's own trade-off, and it is documented in the README as a + setting the user owns. + +### Fail-open + +Every hook body is wrapped. On exception, the hook leaves `output` untouched (all +mutations are staged on clones and committed only after the whole pipeline succeeds) and +logs to `/tokenpilot/adapter.log`. This matters because OpenCode does **not** +catch plugin exceptions. + +## Gaps against the reference (OpenClaw) and how each is handled + +| Gap | Handling | +| :-- | :-- | +| Eviction needed shared primitives | Solved by the preceding refactor PR: the DSH canonical-surface core moved into `@lightrsi/eviction`. OpenCode applies it through a durable overlay. `eviction on/off/set minBlockChars` are exposed as on Claude Code. | +| `dynamicContextTarget=user` | Not supported in 1.18.33 (hook order, above). Runs as `developer`; `doctor` warns. | +| `mode aggressive` | Not exposed, as on Claude Code. Open question for a human. | +| Recovery tool name is prefixed by OpenCode | Same situation as Claude Code (`mcp__…`). The recovery hint names `memory_fault_recover`; the only matching tool is the prefixed one. Verified in the live smoke run (see PR). | +| Reduction is request-local; stored parts keep full output | Same as the Codex/Claude Code proxies. | +| `opencode.jsonc`-only installs | MCP registration is printed, not written, so comments are never destroyed. `doctor` flags it. | +| No in-host slash commands | Shared `lightrsi opencode ...` CLI. | +| `prompt_cache_key` routing | Not emitted. OpenCode already sets its own `promptCacheKey` for providers that support it, and local servers cache by prefix. | +| Context Cleaner | Out of scope. See below. | + +## Context Cleaner (future) + +A v1 plugin can add a `/tokenpilot-clean` command through `command.execute.before` or a +custom `tool`, persist the schedule under the adapter state dir, and apply the selected +cleanup request-locally in the next `messages.transform`. That is the same +"schedule now, apply on next request" shape as DSH's `cleaner-pre-step.ts`, with Claude +Code's request-overlay semantics, because OpenCode has no persistent part-edit hook for +plugins. diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index e721ccec..2608d242 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -195,6 +195,49 @@ importers: specifier: ^5.9.3 version: 5.9.3 + components/adapters/opencode: + dependencies: + '@lightrsi/artifact-store': + specifier: workspace:* + version: link:../../packages/foundation/artifact-store + '@lightrsi/eviction': + specifier: workspace:* + version: link:../../packages/features/eviction + '@lightrsi/history': + specifier: workspace:* + version: link:../../packages/foundation/history + '@lightrsi/host-adapter': + specifier: workspace:* + version: link:../../packages/foundation/host-adapter + '@lightrsi/kernel': + specifier: workspace:* + version: link:../../packages/foundation/kernel + '@lightrsi/mcp': + specifier: workspace:* + version: link:../../products/mcp + '@lightrsi/product-surface': + specifier: workspace:* + version: link:../../packages/foundation/product-surface + '@lightrsi/reduction': + specifier: workspace:* + version: link:../../packages/features/reduction + '@lightrsi/stabilizer': + specifier: workspace:* + version: link:../../packages/features/stabilizer + '@lightrsi/tokenpilot': + specifier: workspace:* + version: link:../../presets/tokenpilot + devDependencies: + esbuild: + specifier: ^0.25.10 + version: 0.25.12 + tsx: + specifier: ^4.21.0 + version: 4.21.0 + typescript: + specifier: ^5.9.3 + version: 5.9.3 + components/packages/features/cleaner: dependencies: '@lightrsi/history': diff --git a/website/.vitepress/config.ts b/website/.vitepress/config.ts index 830cb74f..e768adcf 100644 --- a/website/.vitepress/config.ts +++ b/website/.vitepress/config.ts @@ -113,6 +113,7 @@ export default defineConfig({ { text: 'Codex', link: '/hosts/codex' }, { text: 'Claude Code', link: '/hosts/claude-code' }, { text: 'DeepSeek Harness', link: '/hosts/deepseek-harness' }, + { text: 'OpenCode', link: '/hosts/opencode' }, ], }, ], diff --git a/website/hosts/compatibility.md b/website/hosts/compatibility.md index 9e230e08..781dfcc9 100644 --- a/website/hosts/compatibility.md +++ b/website/hosts/compatibility.md @@ -1,6 +1,6 @@ # Host Compatibility -TokenPilot integrates with OpenClaw, Codex CLI, Claude Code, and DeepSeek Harness. Each integration has a different boundary, so capabilities and setup instructions are documented per host. +TokenPilot integrates with OpenClaw, Codex CLI, Claude Code, DeepSeek Harness, and OpenCode. Each integration has a different boundary, so capabilities and setup instructions are documented per host. ## Supported Hosts @@ -10,6 +10,7 @@ TokenPilot integrates with OpenClaw, Codex CLI, Claude Code, and DeepSeek Harnes | [Codex CLI](./codex) | Local proxy + hooks | `components/adapters/codex/` | | [Claude Code](./claude-code) | Local gateway + MCP | `components/adapters/claude-code/` | | [DeepSeek Harness](./deepseek-harness) | Native Cordis plugin + optional context eviction + session status | `components/adapters/deepseek-harness/` | +| [OpenCode](./opencode) | In-process v1 plugin + recovery MCP | `components/adapters/opencode/` | DeepSeek Harness provides opt-in context eviction and the read-only `/tokenpilot-status` command. It uses Harness's own transport and compaction, with separate profile configuration. See the [DeepSeek Harness guide](./deepseek-harness) for installation, required settings, and troubleshooting. @@ -17,22 +18,24 @@ The matrix below distinguishes shared product interfaces from host-native capabi ## TokenPilot Host Feature Matrix -| Feature | OpenClaw | Codex | Claude Code | DeepSeek Harness | -| :-- | :-- | :-- | :-- | :-- | -| Stable Prefix | ✅ | ✅ | ✅ | — | -| Context Reduction | ✅ | ✅ | ✅ | — | -| Automatic Context Eviction | ✅ | Opt-in response-chain rebase¹ | — | Opt-in; estimator and durable state required | -| User-approved [Context Cleaner](/user-guide/context-cleaner) | Scheduled canonical apply | Scheduled response-chain rebase | Scheduled request overlay | Native `/tokenpilot-clean`; scheduled surface transaction | -| Visual Inspector | ✅ | ✅ | ✅ | — | -| Session Reports (`report`) | ✅ | ✅ | ✅ | —; native status instead | -| Session Status | `/lightrsi status` or CLI | `lightrsi codex status` | `lightrsi claude-code status` | `/tokenpilot-status`² | -| Shared Standalone CLI | ✅ | ✅ | ✅ | —; Harness plugin interface | -| MCP Recovery Server | ✅ | ✅ | ✅ | — | -| `mode conservative` | ✅ | ✅ | ✅ | —; profile configuration | -| `mode normal` | ✅ | ✅ | ✅ | —; profile configuration | -| `mode aggressive` | ✅ | — | — | —; profile configuration | -| Proxy / Gateway Startup | Gateway restart | SessionStart hook | SessionStart hook | Not needed; Harness owns transport | +| Feature | OpenClaw | Codex | Claude Code | DeepSeek Harness | OpenCode | +| :-- | :-- | :-- | :-- | :-- | :-- | +| Stable Prefix | ✅ | ✅ | ✅ | — | ✅ (`developer` target only) | +| Context Reduction | ✅ | ✅ | ✅ | — | ✅ | +| Automatic Context Eviction | ✅ | Opt-in response-chain rebase¹ | — | Opt-in; estimator and durable state required | Opt-in; durable request overlay³ | +| User-approved [Context Cleaner](/user-guide/context-cleaner) | Scheduled canonical apply | Scheduled response-chain rebase | Scheduled request overlay | Native `/tokenpilot-clean`; scheduled surface transaction | — | +| Visual Inspector | ✅ | ✅ | ✅ | — | ✅ | +| Session Reports (`report`) | ✅ | ✅ | ✅ | —; native status instead | ✅ | +| Session Status | `/lightrsi status` or CLI | `lightrsi codex status` | `lightrsi claude-code status` | `/tokenpilot-status`² | `lightrsi opencode status` | +| Shared Standalone CLI | ✅ | ✅ | ✅ | —; Harness plugin interface | ✅ | +| MCP Recovery Server | ✅ | ✅ | ✅ | — | ✅ | +| `mode conservative` | ✅ | ✅ | ✅ | —; profile configuration | ✅ | +| `mode normal` | ✅ | ✅ | ✅ | —; profile configuration | ✅ | +| `mode aggressive` | ✅ | — | — | —; profile configuration | — | +| Proxy / Gateway Startup | Gateway restart | SessionStart hook | SessionStart hook | Not needed; Harness owns transport | Not needed; in-process | ¹ The [Codex adapter README](https://github.com/zjunlp/LightRSI/blob/main/components/adapters/codex/README.md#task-state-estimator-bridge-pr-b) documents opt-in automatic eviction with both `taskStateEstimator.enabled` and `contextRewrite.enabled`. This does not provide the OpenClaw `eviction` commands or `aggressive` mode. ² DeepSeek Harness registers its status command when the profile provides command and session projection services. Its eviction pass runs before native Harness compaction by default. See [DeepSeek Harness](/hosts/deepseek-harness) for configuration and status interpretation. + +³ OpenCode eviction uses the shared canonical-surface core (`@lightrsi/eviction`, lifted from the DeepSeek Harness adapter). It needs `modules.eviction`, `eviction.enabled` and `taskStateEstimator.{baseUrl,model,apiKey}`. Evicted content is archived and recoverable with `memory_fault_recover`. diff --git a/website/hosts/opencode.md b/website/hosts/opencode.md new file mode 100644 index 00000000..41c950bf --- /dev/null +++ b/website/hosts/opencode.md @@ -0,0 +1,45 @@ +# OpenCode + +The OpenCode integration is an **in-process v1 plugin** plus the shared recovery MCP server registered in OpenCode's own config. Verified on OpenCode `1.18.33`. + +## Installation + +```bash +pnpm install +pnpm --filter @lightrsi/mcp build +npm --prefix components/adapters/opencode run build +npm --prefix components/products/cli run build +npm --prefix components/adapters/opencode run install:opencode +``` + +This command, in `~/.config/opencode` (or `$XDG_CONFIG_HOME/opencode`): + +- writes a marker-tagged ESM loader at `plugins/tokenpilot.js` +- adds `mcp.tokenpilot_memory_fault_recover` to `opencode.json` after writing a timestamped backup. An `opencode.jsonc`-only setup is never rewritten; the snippet is printed instead. +- writes `tokenpilot.json` in `normal` mode if it does not exist (never credentials) + +Remove with `npm --prefix components/adapters/opencode run uninstall:opencode`. It removes exactly what install added. + +## How it works + +| TokenPilot feature | OpenCode hook | +| :-- | :-- | +| Stable prefix | `experimental.chat.system.transform` (volatile lines moved to the tail; `developer` target only) | +| Reduction | `experimental.chat.messages.transform`, before every model step | +| Recovery | shared recovery MCP server (`tokenpilot_memory_fault_recover_memory_fault_recover` in OpenCode) | +| Eviction (opt-in) | `experimental.chat.messages.transform` + a durable per-session overlay | + +OpenCode's native tool-output pruning (`compaction.prune`) is off by default in 1.18.33. TokenPilot never changes it, and `doctor` reports its current value. + +## Verification + +1. Restart OpenCode. +2. `lightrsi opencode doctor`: loader, bundle and MCP entry present, MCP probe `ok`. +3. After a few turns: `lightrsi opencode report` + +## Troubleshooting + +- OpenCode does not catch plugin errors; TokenPilot catches its own and logs to `~/.config/opencode/tokenpilot-state/tokenpilot/tokenpilot/adapter.log`. +- `opencode run --pure "…"` runs without plugins for comparison. + +See the [adapter README](https://github.com/zjunlp/LightRSI/blob/main/components/adapters/opencode/README.md) and the [design note](https://github.com/zjunlp/LightRSI/blob/main/docs/adapters/opencode-design.md).