diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a128d19f..1b2a649d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -80,6 +80,9 @@ jobs: - name: Test recovery MCP product run: pnpm --dir components/products/mcp test + - name: Test pi adapter + run: pnpm --dir components/adapters/pi test + - name: Test OpenCode adapter run: pnpm --dir components/adapters/opencode test diff --git a/components/adapters/HOSTS.md b/components/adapters/HOSTS.md index ad2f9cce..7ff64e17 100644 --- a/components/adapters/HOSTS.md +++ b/components/adapters/HOSTS.md @@ -10,6 +10,7 @@ TokenPilot is now structured as a reusable LightRSI component with multiple host | `Codex CLI` | available | hooks + local Responses proxy + shared CLI | `npm --prefix components/adapters/codex run build` then `npm --prefix components/adapters/codex run install:codex` | [codex/README.md](./codex/README.md) | | `Claude Code` | available | gateway routing + observability hooks + shared CLI | `npm --prefix components/adapters/claude-code run build` then `npm --prefix components/adapters/claude-code run install:claude-code` | [claude-code/README.md](./claude-code/README.md) | | `DeepSeek Harness` | available | Cordis plugin + durable projection | `pnpm --filter @lightrsi/deepseek-harness-adapter compatibility:smoke -- --dsh-checkout=/absolute/path/to/deepseek-harness` | [GUA-06](../../docs/acceptance/GUA-06.md) | +| `pi` | available | in-process extension + native recovery tool + shared CLI | `npm --prefix components/adapters/pi run build` then `npm --prefix components/adapters/pi run install:pi` | [pi/README.md](./pi/README.md) | | `OpenCode` | available | in-process v1 plugin + recovery MCP + shared CLI | `npm --prefix components/adapters/opencode run build` then `npm --prefix components/adapters/opencode run install:opencode` | [opencode/README.md](./opencode/README.md) | Each full TokenPilot host adapter binds the versioned preset explicitly. OpenClaw and Codex declare Stabilizer, Reduction, and Eviction; Claude Code currently declares Stabilizer and Reduction. DeepSeek Harness is a separate compatibility adapter with a narrower projection surface. The same adapter-owned product registrations are used by the shared CLI and browser Visual surface where applicable. @@ -22,22 +23,22 @@ Legend: - `partial`: available, but intentionally narrower than the OpenClaw path - `no`: not supported in the current public adapter -| Capability | OpenClaw | Codex CLI | Claude Code | OpenCode | -| :-- | :--: | :--: | :--: | :--: | -| Stable-prefix rewriting | yes | yes | yes | partial (`developer` target only) | -| Before-call reduction | yes | yes | yes | yes | -| Real MCP-backed `memory_fault_recover` | yes | yes | yes | yes | -| Standalone `lightrsi ...` CLI | yes | yes | yes | yes | -| `status` / `doctor` / `report` | yes | yes | yes | yes | -| `visual` | yes | yes | yes | yes | -| `mode conservative` / `mode normal` | yes | yes | yes | yes | -| `mode aggressive` | yes | no | no | no | -| Estimator-driven lifecycle eviction runtime | yes | yes | yes | yes (durable request overlay) | -| Lifecycle eviction controls | yes | no | no | `on` / `off` / `minBlockChars` | -| In-host slash commands | yes | no | no | no | -| Hook-based observability | partial | yes | yes | in-process | -| Local proxy / gateway runtime | yes | yes | yes | not needed (in-process) | -| Session-state / ux-effects persistence | yes | yes | yes | yes | +| Capability | OpenClaw | Codex CLI | Claude Code | pi | OpenCode | +| :-- | :--: | :--: | :--: | :--: | :--: | +| Stable-prefix rewriting | yes | yes | yes | yes | partial (`developer` target only) | +| Before-call reduction | yes | yes | yes | yes | yes | +| Real MCP-backed `memory_fault_recover` | yes | yes | yes | native tool (pi has no MCP) | yes | +| Standalone `lightrsi ...` CLI | yes | yes | yes | yes | yes | +| `status` / `doctor` / `report` | yes | yes | yes | yes | yes | +| `visual` | yes | yes | yes | yes | yes | +| `mode conservative` / `mode normal` | yes | yes | yes | yes | yes | +| `mode aggressive` | yes | no | no | no | no | +| Estimator-driven lifecycle eviction runtime | yes | yes | yes | yes (native `context_edit`) | yes (durable request overlay) | +| Lifecycle eviction controls | yes | no | no | `on` / `off` / `minBlockChars` | `on` / `off` / `minBlockChars` | +| In-host slash commands | yes | no | no | no | no | +| Hook-based observability | partial | yes | yes | in-process | in-process | +| Local proxy / gateway runtime | yes | yes | yes | not needed (in-process) | not needed (in-process) | +| Session-state / ux-effects persistence | yes | yes | yes | yes | yes | ## Host Notes @@ -70,6 +71,14 @@ Legend: - supports TokenPilot status projection and compatibility smoke verification - canonical surface eviction remains opt-in and is guarded by estimator, registry, safety, and revision checks +### pi + +- in-process pi extension (verified on pi 0.87.1); no proxy, no host config mutation +- stable prefix via structured prompt sections (`before_agent_start`), so pi keeps appending section deltas instead of replacing the prompt +- request-time reduction in the `context` hook; recovery as a native tool with the MCP tool's exact schema +- opt-in eviction through native `context_edit` entries at `turn_end`, before pi's own threshold compaction +- design note: [docs/adapters/pi-design.md](../../docs/adapters/pi-design.md) + ### OpenCode - in-process v1 plugin (verified on OpenCode 1.18.33) plus the shared recovery MCP server in `opencode.json` @@ -81,7 +90,7 @@ Legend: ### Shared Visual Surface - `lightrsi visual` now provides a standalone browser visual entrypoint -- the shared visual can switch between `openclaw`, `codex`, `claude-code`, and `opencode` hosts +- the shared visual can switch between `openclaw`, `codex`, `claude-code`, `pi`, and `opencode` hosts - today, the browser visual is backed by snapshot data; OpenClaw still has the richest dataset, while Codex and Claude Code now route their `visual` commands into the shared browser surface ## Boundary diff --git a/components/adapters/README.md b/components/adapters/README.md index cca8e5bf..89b427c4 100644 --- a/components/adapters/README.md +++ b/components/adapters/README.md @@ -23,6 +23,8 @@ Adapter inventory: - adapter for Claude Code - `deepseek-harness/` - Cordis adapter and compatibility smoke path for DeepSeek Harness +- `pi/` + - in-process extension adapter for the pi coding agent - `opencode/` - in-process v1 plugin adapter for OpenCode - `shared/canonical/` @@ -130,4 +132,5 @@ This keeps the first working version small and makes boundary mistakes easier to - [codex/README.md](./codex/README.md) - [claude-code/README.md](./claude-code/README.md) - [deepseek-harness/](./deepseek-harness/) +- [pi/README.md](./pi/README.md) - [opencode/README.md](./opencode/README.md) diff --git a/components/adapters/pi/README.md b/components/adapters/pi/README.md new file mode 100644 index 00000000..aa6f3aee --- /dev/null +++ b/components/adapters/pi/README.md @@ -0,0 +1,137 @@ +# TokenPilot pi Adapter + +This package integrates TokenPilot with [pi](https://pi.dev) (`@earendil-works/pi-coding-agent`, verified on **0.87.1**) as an in-process pi extension. No proxy or gateway is involved: TokenPilot runs inside pi's own request lifecycle. + +This adapter explicitly binds the TokenPilot `stabilizer`, `reduction`, and `eviction` features. Its product registration provides pi state discovery to the shared CLI and Visual surface. + +Design, host facts and gaps: [`docs/adapters/pi-design.md`](../../../docs/adapters/pi-design.md). + +For the component-level overview and shared command surface, see: + +- [`components/presets/tokenpilot/README.md`](../../presets/tokenpilot/README.md) +- [`components/adapters/README.md`](../README.md) +- [`components/adapters/HOSTS.md`](../HOSTS.md) + +## Supports + +| Feature | pi hook | Notes | +| :-- | :-- | :-- | +| Stable prefix | `before_agent_start` | Shared stabilizer over prompt inputs (`appendSystemPrompt`, custom sections, context files). Volatile lines go to a trailing `tokenpilot_dynamic` section (target `developer`, default) or to the first user message (target `user`). Never forces a full system-prompt replacement, so pi keeps appending section deltas. | +| Reduction | `context` | Request-time reduction over the whole history, the same position as the Codex/Claude Code proxies. Tool-result text only; pi's session file keeps the originals. | +| Recovery | native tool `memory_fault_recover` | pi has no MCP. The tool has the same name, description and schema as the shared recovery MCP tool and calls the same resolver. The recovery protocol is added as a stable `tokenpilot_recovery` prompt section. | +| Eviction (opt-in, off by default) | `turn_end` → `context_edit` entries | Estimator-driven lifecycle eviction through the shared canonical-surface core in `@lightrsi/eviction`. Evicted originals are archived and recoverable. `turn_end` entries commit before pi's threshold auto-compaction check, so eviction runs before native compaction. | +| Modes | | `conservative`, `normal` (default). `aggressive` is not exposed. | + +Current limitations: + +- no in-host slash commands; use the shared `lightrsi pi ...` CLI +- the stabilizer is mostly a no-op on pi's default prompt, which has no volatile lines; it acts only on user and project prompt inputs that contain them +- the Context Cleaner is not integrated yet (see the design note) + +## Install + +```bash +pnpm install +npm --prefix components/adapters/pi run build +npm --prefix components/products/cli run build # for the lightrsi CLI +npm --prefix components/adapters/pi run install:pi +``` + +Install writes: + +- `~/.pi/agent/extensions/tokenpilot/index.js`: a marker-tagged loader that `require`s `components/adapters/pi/dist/extension.js`. pi auto-loads it. A pre-existing non-TokenPilot file at that path is moved to `index.js.bak-` first. +- `~/.pi/agent/tokenpilot.json`: runtime config in `normal` mode, created only if missing and never containing credentials. +- `~/.pi/agent/tokenpilot-install.json`: install manifest used by uninstall. +- `lightrsi` launcher in `~/.local/bin` (or `$LIGHTRSI_BIN_DIR`). + +No pi-owned file (`settings.json`, `models.json`, `auth.json`) is modified. `PI_CODING_AGENT_DIR` and `TOKENPILOT_PI_CONFIG` are honoured. + +Uninstall: + +```bash +npm --prefix components/adapters/pi run uninstall:pi # remove the loader, restore any backup +npm --prefix components/adapters/pi run uninstall:pi -- --purge # also remove the created config and state +``` + +## Verify + +1. Start pi, or run `/reload` in a running session, so the extension loads. +2. `lightrsi pi doctor` +3. After a few turns: `lightrsi pi report` + +A healthy doctor shows the loader and bundle present, a writable state dir, `declared features: stabilizer, reduction, eviction`, and the latest session once pi has started one. + +## Commands + +```bash +lightrsi pi status +lightrsi pi report +lightrsi pi doctor +lightrsi pi visual +lightrsi pi mode conservative +lightrsi pi mode normal +lightrsi pi stabilizer on|off +lightrsi pi stabilizer target developer|user +lightrsi pi reduction on|off +lightrsi pi reduction mode light|balanced|aggressive +lightrsi pi reduction pass on|off +lightrsi pi eviction status|on|off +lightrsi pi eviction set minBlockChars +``` + +Reduction passes: `readStateCompaction`, `toolPayloadTrim`, `htmlSlimming`, `execOutputTruncation`, `agentsStartupOptimization`. + +## Configuration + +`~/.pi/agent/tokenpilot.json` uses the same keys and defaults as the Claude Code adapter: + +| Key | Default | Notes | +| :-- | :-- | :-- | +| `enabled` | `true` | Master switch; `false` makes every hook a no-op. | +| `modules.stabilizer` | `true` | | +| `hooks.dynamicContextTarget` | `developer` | `developer` or `user`. | +| `modules.reduction` | `true` | | +| `reduction.triggerMinChars` / `maxToolChars` | `2200` / `1200` | `conservative`: `4000` / `1800`. | +| `reduction.passes.*` | all `true` | | +| `modules.eviction` + `eviction.enabled` | `false` | Both must be on, plus the estimator settings below. | +| `eviction.minBlockChars` | `4000` | | +| `taskStateEstimator.baseUrl` / `model` / `apiKey` | unset | OpenAI-compatible endpoint for the task-state estimator. Keep the key out of checked-in files. | + +Enabling eviction: + +```bash +lightrsi pi eviction on +# then set taskStateEstimator.baseUrl / model / apiKey in ~/.pi/agent/tokenpilot.json +lightrsi pi doctor # "eviction: active" once complete +``` + +## Runtime Files + +```text +~/.pi/agent/tokenpilot-state/tokenpilot/ +``` + +- `tokenpilot/adapter.log`: fail-open reasons and debug lines. Nothing is written to pi's terminal. +- `tokenpilot/tool-result-archives//`: originals of trimmed and evicted content (recovery) +- `tokenpilot/reduction-memo/.json`: which read disclosed which file, so a restarted pi (`pi -p`, `--continue`) still honours deliberate re-reads +- `session-state/latest.json`, `session-state/bindings/.jsonl` +- `ux-effects/latest.json`, `ux-effects/sessions/.json` +- the registry under the shared history layout (eviction task state, when enabled) + +## Debugging + +- `lightrsi pi doctor` lists every problem it finds. +- `tail ~/.pi/agent/tokenpilot-state/tokenpilot/tokenpilot/adapter.log`: every hook that failed open logs ` failed open `. pi then continues with its own, unmodified data. +- Set `"logLevel": "debug"` in `tokenpilot.json` for eviction readiness details. +- To confirm the extension loaded, run pi with `--verbose` and look for the `tokenpilot` extension. + +## Package Scripts + +```bash +npm --prefix components/adapters/pi run build +npm --prefix components/adapters/pi run typecheck +npm --prefix components/adapters/pi test +npm --prefix components/adapters/pi run install:pi +npm --prefix components/adapters/pi run uninstall:pi +npm --prefix components/adapters/pi run doctor:pi +``` diff --git a/components/adapters/pi/build.ts b/components/adapters/pi/build.ts new file mode 100644 index 00000000..92b6f356 --- /dev/null +++ b/components/adapters/pi/build.ts @@ -0,0 +1,31 @@ +import { build } from "esbuild"; +import { join } from "node:path"; + +async function main() { + await build({ + entryPoints: { + extension: "src/extension.ts", + "install-pi": "scripts/install-pi.ts", + "uninstall-pi": "scripts/uninstall-pi.ts", + "doctor-pi": "scripts/doctor-pi.ts", + }, + bundle: true, + // ../shared/canonical has no node_modules of its own; resolve workspace deps from here. + nodePaths: [join(__dirname, "node_modules")], + outdir: "dist", + platform: "node", + target: "node20", + format: "cjs", + sourcemap: true, + minify: false, + logLevel: "info", + logOverride: { + "empty-import-meta": "silent", + }, + }); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/components/adapters/pi/package.json b/components/adapters/pi/package.json new file mode 100644 index 00000000..12522190 --- /dev/null +++ b/components/adapters/pi/package.json @@ -0,0 +1,44 @@ +{ + "name": "@lightrsi/pi-adapter", + "version": "0.1.0-beta.1", + "description": "TokenPilot adapter for the pi coding agent (in-process extension).", + "lightrsi": { + "role": "adapter" + }, + "main": "dist/extension.js", + "license": "MIT", + "repository": { + "type": "git", + "url": "https://github.com/zjunlp/LightRSI.git", + "directory": "components/adapters/pi" + }, + "engines": { + "node": ">=20.0.0" + }, + "dependencies": { + "@lightrsi/artifact-store": "workspace:*", + "@lightrsi/eviction": "workspace:*", + "@lightrsi/history": "workspace:*", + "@lightrsi/host-adapter": "workspace:*", + "@lightrsi/kernel": "workspace:*", + "@lightrsi/mcp": "workspace:*", + "@lightrsi/product-surface": "workspace:*", + "@lightrsi/reduction": "workspace:*", + "@lightrsi/stabilizer": "workspace:*", + "@lightrsi/tokenpilot": "workspace:*" + }, + "scripts": { + "build": "tsx build.ts", + "doctor:pi": "node --import tsx scripts/doctor-pi.ts", + "install:pi": "node --import tsx scripts/install-pi.ts", + "uninstall:pi": "node --import tsx scripts/uninstall-pi.ts", + "test": "node --import tsx --test tests/*.test.ts", + "typecheck": "tsc -p tsconfig.json --noEmit" + }, + "devDependencies": { + "esbuild": "^0.25.10", + "tsx": "^4.21.0", + "typescript": "^5.9.3" + }, + "keywords": ["pi", "tokenpilot", "context-management", "extension"] +} diff --git a/components/adapters/pi/scripts/doctor-pi.ts b/components/adapters/pi/scripts/doctor-pi.ts new file mode 100644 index 00000000..13c9ace7 --- /dev/null +++ b/components/adapters/pi/scripts/doctor-pi.ts @@ -0,0 +1,16 @@ +#!/usr/bin/env node +import { defaultTokenPilotPiConfigPath, loadTokenPilotPiConfig } from "../src/config.js"; +import { formatPiDoctorReport, inspectPiDoctor } from "../src/doctor.js"; + +async function main() { + const configPath = process.env.TOKENPILOT_PI_CONFIG ?? defaultTokenPilotPiConfigPath(); + const config = await loadTokenPilotPiConfig(configPath); + const report = await inspectPiDoctor({ config, configPath, agentDir: process.env.PI_CODING_AGENT_DIR }); + console.log(formatPiDoctorReport(report)); + if (!report.healthy) process.exitCode = 1; +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); +}); diff --git a/components/adapters/pi/scripts/install-pi.ts b/components/adapters/pi/scripts/install-pi.ts new file mode 100644 index 00000000..c2985751 --- /dev/null +++ b/components/adapters/pi/scripts/install-pi.ts @@ -0,0 +1,29 @@ +#!/usr/bin/env node +import { installPiTokenPilot } from "../src/install.js"; + +async function main() { + const result = await installPiTokenPilot({ + agentDir: process.env.PI_CODING_AGENT_DIR, + configPath: process.env.TOKENPILOT_PI_CONFIG, + }); + console.log([ + "TokenPilot pi install complete:", + `- extension loader: ${result.paths.loaderPath}`, + `- extension bundle: ${result.bundlePath}`, + ...(result.backupPath ? [`- previous loader backed up to: ${result.backupPath}`] : []), + `- tokenpilot config: ${result.paths.configPath} (${result.configCreated ? "created, normal mode" : "kept existing"})`, + `- state dir: ${result.stateDir}`, + `- install manifest: ${result.paths.manifestPath}`, + result.cliBin?.installed + ? `- lightrsi CLI bin: installed at ${result.cliBin.launcherPath ?? result.cliBin.binPath}` + : `- lightrsi CLI bin: skipped (build components/products/cli first)`, + ...(result.cliBin && !result.cliBin.binDirOnPath ? [`- lightrsi CLI PATH note: add ${result.cliBin.binDir} to PATH if 'lightrsi' is unavailable.`] : []), + "- next step: start (or /reload) pi so the extension loads", + "- verify: lightrsi pi doctor", + ].join("\n")); +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); +}); diff --git a/components/adapters/pi/scripts/uninstall-pi.ts b/components/adapters/pi/scripts/uninstall-pi.ts new file mode 100644 index 00000000..5320602c --- /dev/null +++ b/components/adapters/pi/scripts/uninstall-pi.ts @@ -0,0 +1,24 @@ +#!/usr/bin/env node +import { uninstallPiTokenPilot } from "../src/install.js"; + +async function main() { + const purge = process.argv.includes("--purge"); + const result = await uninstallPiTokenPilot({ + agentDir: process.env.PI_CODING_AGENT_DIR, + configPath: process.env.TOKENPILOT_PI_CONFIG, + purge, + }); + console.log([ + "TokenPilot pi uninstall:", + `- loader removed: ${result.loaderRemoved ? "yes" : "no"}`, + `- previous loader restored: ${result.backupRestored ? "yes" : "no"}`, + ...(result.foreignLoaderKept ? ["- a non-TokenPilot loader exists at the path and was left untouched"] : []), + `- config removed: ${result.configRemoved ? "yes" : purge ? "no (not created by install)" : "no (pass --purge)"}`, + `- state removed: ${result.stateRemoved ? "yes" : purge ? "no" : "no (pass --purge)"}`, + ].join("\n")); +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); +}); diff --git a/components/adapters/pi/src/codec.ts b/components/adapters/pi/src/codec.ts new file mode 100644 index 00000000..be93d9fc --- /dev/null +++ b/components/adapters/pi/src/codec.ts @@ -0,0 +1,197 @@ +/** + * pi transcript bridge: `AgentMessage[]` ↔ canonical `RuntimeMessage[]`. + * + * Decoding is lossless for everything TokenPilot may rewrite (tool-result text, + * first-user-message text) and carries everything else by reference: encoding + * starts from the original pi objects and clones only the messages whose + * canonical text changed. An untouched envelope therefore round-trips to the + * identical array of identical objects. + * + * Mapping (pi 0.87.1): + * user → user (text blocks; images → image blocks) + * assistant → assistant (text → text, toolCall → tool_call; thinking kept in metadata) + * toolResult → tool (one tool_result block; image parts counted in metadata) + * bashExecution, custom, branchSummary, compactionSummary, unknown + * → user pass-through (visible text only; never rewritten) + */ +import type { RuntimeContentBlock, RuntimeMessage } from "@lightrsi/kernel"; +import type { CanonicalSurfaceEntry } from "../../shared/canonical/eviction.js"; +import type { + PiAgentMessage, + PiAssistantMessage, + PiImageContent, + PiProjectedSessionEntry, + PiTextContent, + PiToolResultMessage, + PiUserMessage, +} from "./pi-types.js"; + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function textOfParts(parts: unknown): string { + if (typeof parts === "string") return parts; + if (!Array.isArray(parts)) return ""; + return parts + .map((part) => (isRecord(part) && part.type === "text" && typeof part.text === "string" ? part.text : "")) + .filter(Boolean) + .join("\n"); +} + +function passThroughText(message: Record): string { + switch (message.role) { + case "bashExecution": + return [`$ ${String(message.command ?? "")}`, String(message.output ?? "")].join("\n").trim(); + case "custom": + return textOfParts(message.content); + case "branchSummary": + case "compactionSummary": + return String(message.summary ?? ""); + default: + return textOfParts(message.content); + } +} + +export function decodePiMessage(message: PiAgentMessage, index: number): RuntimeMessage { + const meta = { piIndex: index, piRole: isRecord(message) ? String(message.role ?? "") : "" }; + if (!isRecord(message)) { + return { role: "user", content: "", metadata: { ...meta, lightrsiPassThrough: true } }; + } + if (message.role === "user") { + const user = message as PiUserMessage; + if (typeof user.content === "string") return { role: "user", content: user.content, metadata: meta }; + const blocks: RuntimeContentBlock[] = []; + for (const part of Array.isArray(user.content) ? user.content : []) { + if (!isRecord(part)) continue; + if (part.type === "text" && typeof part.text === "string") blocks.push({ type: "text", text: part.text }); + else if (part.type === "image") blocks.push({ type: "image", mediaType: String((part as PiImageContent).mimeType ?? "") }); + } + return { role: "user", content: blocks, metadata: meta }; + } + if (message.role === "assistant") { + const assistant = message as PiAssistantMessage; + const blocks: RuntimeContentBlock[] = []; + const reasoning: string[] = []; + for (const part of Array.isArray(assistant.content) ? assistant.content : []) { + if (!isRecord(part)) continue; + if (part.type === "text" && typeof part.text === "string") blocks.push({ type: "text", text: part.text }); + else if (part.type === "toolCall" && typeof part.id === "string") { + blocks.push({ + type: "tool_call", + toolCallId: part.id, + toolName: String(part.name ?? ""), + argumentsJson: isRecord(part.arguments) ? part.arguments : undefined, + }); + } else if (part.type === "thinking" && typeof part.thinking === "string") { + reasoning.push(part.thinking); + } + } + return { role: "assistant", content: blocks, metadata: { ...meta, ...(reasoning.length > 0 ? { reasoning } : {}) } }; + } + if (message.role === "toolResult") { + const result = message as PiToolResultMessage; + const parts = Array.isArray(result.content) ? result.content : []; + return { + role: "tool", + content: [{ + type: "tool_result", + toolCallId: typeof result.toolCallId === "string" ? result.toolCallId : undefined, + toolName: typeof result.toolName === "string" ? result.toolName : undefined, + status: result.isError === true ? "error" : "success", + text: textOfParts(parts), + }], + metadata: { ...meta, imageParts: parts.filter((part) => isRecord(part) && part.type === "image").length }, + }; + } + return { role: "user", content: passThroughText(message), metadata: { ...meta, lightrsiPassThrough: true } }; +} + +export function decodePiMessages(messages: readonly PiAgentMessage[]): RuntimeMessage[] { + return messages.map((message, index) => decodePiMessage(message, index)); +} + +function firstText(content: RuntimeMessage["content"]): string { + if (typeof content === "string") return content; + return content + .flatMap((block) => (block.type === "text" || block.type === "tool_result" ? [block.text] : [])) + .join("\n"); +} + +/** + * Apply canonical changes back onto the original pi messages. Only tool-result + * text and user text are writable; everything else is taken from `original`. + */ +export function encodePiMessages( + original: readonly PiAgentMessage[], + canonical: readonly RuntimeMessage[], +): PiAgentMessage[] { + const decodedOriginal = decodePiMessages(original); + return original.map((message, index) => { + const next = canonical[index]; + const before = decodedOriginal[index]; + if (!next || !before || !isRecord(message) || next.metadata?.lightrsiPassThrough === true) return message; + const nextText = firstText(next.content); + if (nextText === firstText(before.content)) return message; + + if (message.role === "toolResult") { + const result = message as PiToolResultMessage; + const images = (Array.isArray(result.content) ? result.content : []).filter((part) => isRecord(part) && part.type !== "text"); + return { ...result, content: [{ type: "text", text: nextText } as PiTextContent, ...images] }; + } + if (message.role === "user") { + const user = message as PiUserMessage; + if (typeof user.content === "string") return { ...user, content: nextText }; + const parts = Array.isArray(user.content) ? user.content : []; + const images = parts.filter((part) => isRecord(part) && part.type !== "text"); + return { ...user, content: [{ type: "text", text: nextText } as PiTextContent, ...images] }; + } + return message; + }); +} + +/* ------------------------------------------------------------------ * + * Session projection → canonical surface entries (eviction) + * ------------------------------------------------------------------ */ + +const EVICTION_STUB_PREFIX = "[evicted:"; + +/** + * Turn the projected branch (`turn_end.context.contextEntries`) into canonical + * surface entries. Message entries become items; compaction, branch-summary and + * custom-message entries become checkpoints; state-only entries are skipped. + * Turn N starts at the Nth `user` message. + */ +export function piProjectionToSurfaceEntries(entries: readonly PiProjectedSessionEntry[]): CanonicalSurfaceEntry[] { + const out: CanonicalSurfaceEntry[] = []; + let turn = 0; + entries.forEach((projected, entryIndex) => { + const source = projected?.sourceEntry; + if (!source || typeof source.id !== "string" || !Array.isArray(projected.messages)) return; + if (projected.messages.length === 0) return; + const isMessageEntry = source.type === "message" && projected.messages.length === 1; + projected.messages.forEach((message, messageIndex) => { + if (!isRecord(message) || message.role === "system") return; + if (message.role === "user" && isMessageEntry) turn += 1; + const canonical = decodePiMessage(message, entryIndex); + const id = isMessageEntry ? source.id : `${source.id}#${messageIndex}`; + out.push({ + id, + turn, + message: canonical, + checkpoint: !isMessageEntry || canonical.metadata?.lightrsiPassThrough === true, + alreadyEvicted: firstText(canonical.content).startsWith(EVICTION_STUB_PREFIX), + }); + }); + }); + return out; +} + +/** Replacement content for a `context_edit`, shaped for the target's role. */ +export function piReplacementContent( + message: PiAgentMessage | undefined, + text: string, +): string | PiTextContent[] { + if (isRecord(message) && message.role === "user" && typeof message.content === "string") return text; + return [{ type: "text", text }]; +} diff --git a/components/adapters/pi/src/config.ts b/components/adapters/pi/src/config.ts new file mode 100644 index 00000000..dc755f68 --- /dev/null +++ b/components/adapters/pi/src/config.ts @@ -0,0 +1,45 @@ +import { dirname, join, resolve } from "node:path"; +import { + loadCanonicalAdapterConfig, + normalizeCanonicalAdapterConfig, + runtimeHomeDir, + writeCanonicalAdapterConfig, + type CanonicalAdapterConfig, +} from "../../shared/canonical/config.js"; + +export type TokenPilotPiConfig = CanonicalAdapterConfig; + +/** pi's agent directory (`~/.pi/agent`, override `PI_CODING_AGENT_DIR`). */ +export function defaultPiAgentDir(): string { + const override = process.env.PI_CODING_AGENT_DIR?.trim(); + return override ? resolve(override) : join(runtimeHomeDir(), ".pi", "agent"); +} + +export function defaultTokenPilotPiConfigPath(): string { + return process.env.TOKENPILOT_PI_CONFIG + ? resolve(process.env.TOKENPILOT_PI_CONFIG) + : join(defaultPiAgentDir(), "tokenpilot.json"); +} + +export function defaultPiStateDir(configPath = defaultTokenPilotPiConfigPath()): string { + return join(dirname(configPath), "tokenpilot-state", "tokenpilot"); +} + +/** Directory pi auto-loads user extensions from. */ +export function defaultPiExtensionDir(): string { + return join(defaultPiAgentDir(), "extensions", "tokenpilot"); +} + +export function normalizeTokenPilotPiConfig(raw: unknown, options?: { configPath?: string }): TokenPilotPiConfig { + return normalizeCanonicalAdapterConfig(raw, { + defaultStateDir: defaultPiStateDir(options?.configPath ?? defaultTokenPilotPiConfigPath()), + }); +} + +export function loadTokenPilotPiConfig(configPath = defaultTokenPilotPiConfigPath()): Promise { + return loadCanonicalAdapterConfig(configPath, { defaultStateDir: defaultPiStateDir(configPath) }); +} + +export function writeTokenPilotPiConfig(config: TokenPilotPiConfig, configPath = defaultTokenPilotPiConfigPath()): Promise { + return writeCanonicalAdapterConfig(config, configPath); +} diff --git a/components/adapters/pi/src/doctor.ts b/components/adapters/pi/src/doctor.ts new file mode 100644 index 00000000..815ac081 --- /dev/null +++ b/components/adapters/pi/src/doctor.ts @@ -0,0 +1,142 @@ +/** + * `lightrsi pi doctor` / `npm run doctor:pi`: verifies the install and runtime state. + * Every check is local and read-only. + */ +import { spawnSync } from "node:child_process"; +import { accessSync, constants, existsSync, readFileSync } from "node:fs"; +import { readLatestUxEffect, resolveLatestSessionId } from "@lightrsi/host-adapter"; +import { canonicalRecoveryProtocolText } from "../../shared/canonical/before-call.js"; +import { evaluateEvictionReadiness, type EvictionReadiness } from "../../shared/canonical/config.js"; +import { adapterLogPath } from "../../shared/canonical/logger.js"; +import type { TokenPilotPiConfig } from "./config.js"; +import { isTokenPilotPiLoader, resolvePiInstallPaths } from "./install.js"; +import { PI_TOKENPILOT_HOST_BINDING, piSupportsEviction } from "./preset.js"; + +export type PiDoctorReport = { + configPath: string; + configExists: boolean; + enabled: boolean; + loaderPath: string; + loaderInstalled: boolean; + loaderBundlePath?: string; + bundleExists: boolean; + stateDir: string; + stateDirWritable: boolean; + declaredFeatures: readonly string[]; + stabilizer: boolean; + dynamicContextTarget: string; + reduction: boolean; + recoveryTool: "native tool (pi has no MCP)"; + recoveryProtocolInjected: boolean; + eviction: EvictionReadiness; + piVersion?: string; + latestSessionId?: string; + latestActivityAt?: string; + adapterLogExists: boolean; + healthy: boolean; + problems: string[]; +}; + +/** The bundle path the installed loader requires (`require("")`). */ +export function loaderTarget(loaderPath: string): string | undefined { + try { + const literal = /require\(("(?:[^"\\]|\\.)*")\)/.exec(readFileSync(loaderPath, "utf8"))?.[1]; + return literal ? JSON.parse(literal) as string : undefined; + } catch { + return undefined; + } +} + +function writable(path: string): boolean { + try { + accessSync(path, constants.W_OK); + return true; + } catch { + return false; + } +} + +function detectPiVersion(): string | undefined { + try { + const out = spawnSync("pi", ["--version"], { encoding: "utf8", timeout: 5000 }); + const text = `${out.stdout ?? ""}`.trim(); + return out.status === 0 && text ? text.split("\n")[0] : undefined; + } catch { + return undefined; + } +} + +export async function inspectPiDoctor(params: { + config: TokenPilotPiConfig; + configPath: string; + agentDir?: string; + detectVersion?: boolean; +}): Promise { + const paths = resolvePiInstallPaths({ agentDir: params.agentDir, configPath: params.configPath }); + const loaderInstalled = await isTokenPilotPiLoader(paths.loaderPath); + const loaderBundlePath = loaderInstalled ? loaderTarget(paths.loaderPath) : undefined; + const bundleExists = Boolean(loaderBundlePath && existsSync(loaderBundlePath)); + const stateDir = params.config.stateDir; + const stateDirWritable = existsSync(stateDir) && writable(stateDir); + const eviction = evaluateEvictionReadiness(params.config, piSupportsEviction()); + const latestSessionId = stateDir ? await resolveLatestSessionId(stateDir).catch(() => undefined) : undefined; + const latest = stateDir ? await readLatestUxEffect(stateDir).catch(() => null) : null; + + const problems: string[] = []; + const configExists = existsSync(params.configPath); + if (!configExists) problems.push(`config missing: ${params.configPath} (run install:pi)`); + if (!params.config.enabled) problems.push("adapter disabled in config (enabled=false)"); + if (!loaderInstalled) problems.push(`pi loader missing: ${paths.loaderPath} (run install:pi)`); + if (loaderInstalled && !bundleExists) problems.push(`loader points to a missing bundle: ${loaderBundlePath ?? "(unparseable)"} (run build)`); + if (!stateDirWritable) problems.push(`state dir not writable: ${stateDir}`); + if (params.config.modules.eviction && !eviction.active) problems.push(`eviction requested but inactive: ${eviction.reason}${eviction.missing.length ? ` (missing ${eviction.missing.join(", ")})` : ""}`); + + return { + configPath: params.configPath, + configExists, + enabled: params.config.enabled, + loaderPath: paths.loaderPath, + loaderInstalled, + ...(loaderBundlePath ? { loaderBundlePath } : {}), + bundleExists, + stateDir, + stateDirWritable, + declaredFeatures: PI_TOKENPILOT_HOST_BINDING.supportedFeatures, + stabilizer: params.config.modules.stabilizer, + dynamicContextTarget: params.config.hooks.dynamicContextTarget, + reduction: params.config.modules.reduction, + recoveryTool: "native tool (pi has no MCP)", + recoveryProtocolInjected: canonicalRecoveryProtocolText().length > 0, + eviction, + ...(params.detectVersion === false ? {} : { piVersion: detectPiVersion() }), + ...(latestSessionId ? { latestSessionId } : {}), + ...(typeof latest?.at === "string" ? { latestActivityAt: latest.at } : {}), + adapterLogExists: existsSync(adapterLogPath(stateDir)), + healthy: problems.length === 0, + problems, + }; +} + +const yesNo = (value: boolean) => (value ? "yes" : "no"); + +export function formatPiDoctorReport(report: PiDoctorReport): string { + return [ + "TokenPilot pi doctor:", + `- tokenpilot config: ${report.configPath} (${report.configExists ? "present" : "missing"})`, + `- enabled: ${yesNo(report.enabled)}`, + `- pi version: ${report.piVersion ?? "(pi not found on PATH)"}`, + `- extension loader: ${report.loaderPath} (${report.loaderInstalled ? "installed" : "missing"})`, + `- extension bundle: ${report.loaderBundlePath ?? "(unknown)"} (${report.bundleExists ? "present" : "missing"})`, + `- stateDir: ${report.stateDir} (${report.stateDirWritable ? "writable" : "not writable"})`, + `- declared features: ${report.declaredFeatures.join(", ")}`, + `- stabilizer: ${yesNo(report.stabilizer)} (target ${report.dynamicContextTarget})`, + `- reduction: ${yesNo(report.reduction)}`, + `- recovery: ${report.recoveryTool}; protocol injected: ${yesNo(report.recoveryProtocolInjected)}`, + `- eviction: ${report.eviction.active ? "active" : `inactive (${report.eviction.reason})`}`, + `- latest session: ${report.latestSessionId ?? "(none yet — start a pi session)"}`, + `- latest activity: ${report.latestActivityAt ?? "(none yet)"}`, + `- adapter log: ${report.adapterLogExists ? adapterLogPath(report.stateDir) : "(none)"}`, + `- healthy: ${yesNo(report.healthy)}`, + ...report.problems.map((problem) => `- problem: ${problem}`), + ].join("\n"); +} diff --git a/components/adapters/pi/src/extension.ts b/components/adapters/pi/src/extension.ts new file mode 100644 index 00000000..a2f1c600 --- /dev/null +++ b/components/adapters/pi/src/extension.ts @@ -0,0 +1,24 @@ +/** + * pi extension entry point (`~/.pi/agent/extensions/tokenpilot/index.js` loads this). + * + * The factory only registers handlers and the recovery tool. Config, state roots + * and caches are set up in `session_start`, as pi's extension contract requires + * (no long-lived resources in the factory). + */ +import type { PiExtensionAPI } from "./pi-types.js"; +import { PiTokenPilotRuntime, type PiRuntimeDependencies } from "./runtime.js"; + +export function registerTokenPilotPiExtension(pi: PiExtensionAPI, deps?: PiRuntimeDependencies): PiTokenPilotRuntime { + const runtime = new PiTokenPilotRuntime(deps); + pi.on("session_start", (event, ctx) => runtime.onSessionStart(event, ctx)); + pi.on("session_shutdown", () => runtime.onSessionShutdown()); + pi.on("before_agent_start", (event, ctx) => runtime.onBeforeAgentStart(event, ctx)); + pi.on("context", (event, ctx) => runtime.onContext(event, ctx)); + pi.on("turn_end", (event, ctx) => runtime.onTurnEnd(event, ctx)); + pi.registerTool(runtime.recoveryTool()); + return runtime; +} + +export default function tokenPilotPiExtension(pi: PiExtensionAPI): void { + registerTokenPilotPiExtension(pi); +} diff --git a/components/adapters/pi/src/host-config-adapter.ts b/components/adapters/pi/src/host-config-adapter.ts new file mode 100644 index 00000000..3cbad5e7 --- /dev/null +++ b/components/adapters/pi/src/host-config-adapter.ts @@ -0,0 +1,15 @@ +import type { ProductSurfaceConfigAdapter } from "@lightrsi/host-adapter"; + +export function resolvePiStateDir(config: Record): string | undefined { + const stateDir = config.stateDir; + return typeof stateDir === "string" && stateDir.trim().length > 0 ? stateDir.trim() : undefined; +} + +/** pi's runtime config is flat, like Claude Code's `tokenpilot.json`. */ +export const piProductSurfaceConfigAdapter: ProductSurfaceConfigAdapter = { + pluginConfigRecord: (config) => config, + pluginEntryRecord: (config) => config, + ensurePluginConfig: (config) => config, + ensurePluginEntry: (config) => config, + resolveStateDir: resolvePiStateDir, +}; diff --git a/components/adapters/pi/src/install.ts b/components/adapters/pi/src/install.ts new file mode 100644 index 00000000..25cfe87f --- /dev/null +++ b/components/adapters/pi/src/install.ts @@ -0,0 +1,222 @@ +/** + * Reversible install for the pi adapter. + * + * install: writes a marker-tagged loader at `/extensions/tokenpilot/index.js` + * that requires the built `dist/extension.js`, and writes + * `/tokenpilot.json` in normal mode if it does not exist yet. + * No pi-owned file is modified. A pre-existing non-TokenPilot file at the + * loader path is moved to a timestamped backup first. + * uninstall: removes the loader only if it carries the marker, restores the backup + * recorded at install time, and (with `purge`) removes the config and + * state this install created. Credentials are never written by install. + */ +import { existsSync } from "node:fs"; +import { mkdir, readFile, readdir, rename, rm, rmdir, writeFile } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { installLightRsiCliBin } from "../../shared/cli-bin-install.js"; +import { rememberCliHostPathOverrides } from "../../shared/cli-context.js"; +import { + defaultPiAgentDir, + defaultTokenPilotPiConfigPath, + loadTokenPilotPiConfig, + normalizeTokenPilotPiConfig, + writeTokenPilotPiConfig, +} from "./config.js"; + +export const PI_LOADER_MARKER = "// tokenpilot-managed-loader: pi"; + +export type PiInstallManifest = { + version: 1; + loaderPath: string; + bundlePath: string; + backupPath?: string; + configPath: string; + configCreated: boolean; + installedAt: string; +}; + +export type PiInstallPaths = { + agentDir: string; + loaderDir: string; + loaderPath: string; + configPath: string; + manifestPath: string; +}; + +export function resolvePiInstallPaths(params?: { agentDir?: string; configPath?: string }): PiInstallPaths { + const agentDir = params?.agentDir ? resolve(params.agentDir) : defaultPiAgentDir(); + const loaderDir = join(agentDir, "extensions", "tokenpilot"); + return { + agentDir, + loaderDir, + loaderPath: join(loaderDir, "index.js"), + configPath: params?.configPath ? resolve(params.configPath) : (params?.agentDir ? join(agentDir, "tokenpilot.json") : defaultTokenPilotPiConfigPath()), + manifestPath: join(agentDir, "tokenpilot-install.json"), + }; +} + +export function piAdapterRoot(moduleDir = __dirname): string { + let current = resolve(moduleDir); + for (let i = 0; i < 6; i += 1) { + if (existsSync(join(current, "package.json")) && existsSync(join(current, "src", "extension.ts"))) return current; + current = dirname(current); + } + return resolve(moduleDir, ".."); +} + +/** + * pi imports extensions with jiti (`import(path, { default: true })`) and requires the + * resulting value to be the factory function itself (pi 0.87.1 `loadExtensionModule`). + * The loader therefore exports the factory as the module value, with `.default` + * pointing back at it, so every interop mode yields a function. + */ +export function renderPiLoader(bundlePath: string): string { + return [ + PI_LOADER_MARKER, + "// Installed by `npm --prefix components/adapters/pi run install:pi`. Remove with `uninstall:pi`.", + `const bundle = require(${JSON.stringify(bundlePath)});`, + "const factory = typeof bundle === \"function\" ? bundle : bundle.default;", + "module.exports = factory;", + "module.exports.default = factory;", + "", + ].join("\n"); +} + +export async function isTokenPilotPiLoader(path: string): Promise { + if (!existsSync(path)) return false; + try { + return (await readFile(path, "utf8")).startsWith(PI_LOADER_MARKER); + } catch { + return false; + } +} + +async function readManifest(path: string): Promise { + try { + const parsed = JSON.parse(await readFile(path, "utf8")) as PiInstallManifest; + return parsed && parsed.version === 1 ? parsed : undefined; + } catch { + return undefined; + } +} + +export async function installPiTokenPilot(params?: { + agentDir?: string; + configPath?: string; + bundlePath?: string; + cliBinDir?: string; + cliContextPath?: string; + installCliBin?: boolean; + now?: () => Date; +}): Promise<{ + paths: PiInstallPaths; + bundlePath: string; + loaderWritten: boolean; + backupPath?: string; + configCreated: boolean; + stateDir: string; + cliBin?: Awaited>; +}> { + const paths = resolvePiInstallPaths(params); + const now = params?.now ?? (() => new Date()); + const bundlePath = params?.bundlePath ? resolve(params.bundlePath) : join(piAdapterRoot(), "dist", "extension.js"); + if (!existsSync(bundlePath)) { + throw new Error(`pi adapter bundle not built: ${bundlePath}. Run \`npm --prefix components/adapters/pi run build\` first.`); + } + + const previous = await readManifest(paths.manifestPath); + let backupPath = previous?.backupPath; + if (existsSync(paths.loaderPath) && !(await isTokenPilotPiLoader(paths.loaderPath))) { + backupPath = `${paths.loaderPath}.bak-${now().getTime()}`; + await rename(paths.loaderPath, backupPath); + } + await mkdir(paths.loaderDir, { recursive: true }); + await writeFile(paths.loaderPath, renderPiLoader(bundlePath), "utf8"); + + const configCreated = !existsSync(paths.configPath); + const config = configCreated + ? normalizeTokenPilotPiConfig({}, { configPath: paths.configPath }) + : await loadTokenPilotPiConfig(paths.configPath); + if (configCreated) await writeTokenPilotPiConfig(config, paths.configPath); + await mkdir(config.stateDir, { recursive: true }); + + const manifest: PiInstallManifest = { + version: 1, + loaderPath: paths.loaderPath, + bundlePath, + ...(backupPath ? { backupPath } : {}), + configPath: paths.configPath, + configCreated: configCreated || previous?.configCreated === true, + installedAt: now().toISOString(), + }; + await writeFile(paths.manifestPath, `${JSON.stringify(manifest, null, 2)}\n`, "utf8"); + + await rememberCliHostPathOverrides("pi", { tokenPilotConfigPath: paths.configPath }, params?.cliContextPath); + const cliBin = params?.installCliBin === false + ? undefined + : await installLightRsiCliBin({ adapterRoot: piAdapterRoot(), binDir: params?.cliBinDir }); + + return { + paths, + bundlePath, + loaderWritten: true, + ...(backupPath ? { backupPath } : {}), + configCreated, + stateDir: config.stateDir, + ...(cliBin ? { cliBin } : {}), + }; +} + +export async function uninstallPiTokenPilot(params?: { + agentDir?: string; + configPath?: string; + purge?: boolean; +}): Promise<{ + loaderRemoved: boolean; + backupRestored: boolean; + foreignLoaderKept: boolean; + configRemoved: boolean; + stateRemoved: boolean; +}> { + const paths = resolvePiInstallPaths(params); + const manifest = await readManifest(paths.manifestPath); + let loaderRemoved = false; + let foreignLoaderKept = false; + if (await isTokenPilotPiLoader(paths.loaderPath)) { + await rm(paths.loaderPath); + loaderRemoved = true; + } else if (existsSync(paths.loaderPath)) { + foreignLoaderKept = true; + } + + let backupRestored = false; + if (manifest?.backupPath && existsSync(manifest.backupPath) && !existsSync(paths.loaderPath)) { + await rename(manifest.backupPath, paths.loaderPath); + backupRestored = true; + } + if (existsSync(paths.loaderDir) && (await readdir(paths.loaderDir)).length === 0) { + await rmdir(paths.loaderDir); + } + + let configRemoved = false; + let stateRemoved = false; + if (params?.purge) { + const configPath = manifest?.configPath ?? paths.configPath; + const stateDir = existsSync(configPath) ? (await loadTokenPilotPiConfig(configPath)).stateDir : undefined; + if (manifest?.configCreated && existsSync(configPath)) { + await rm(configPath); + configRemoved = true; + } + if (stateDir && existsSync(stateDir)) { + await rm(stateDir, { recursive: true, force: true }); + stateRemoved = true; + // Default layout is /tokenpilot-state/tokenpilot; drop the parent if now empty. + const parent = dirname(stateDir); + if (parent.endsWith("tokenpilot-state") && existsSync(parent) && (await readdir(parent)).length === 0) { + await rmdir(parent); + } + } + } + await rm(paths.manifestPath, { force: true }); + return { loaderRemoved, backupRestored, foreignLoaderKept, configRemoved, stateRemoved }; +} diff --git a/components/adapters/pi/src/pi-types.ts b/components/adapters/pi/src/pi-types.ts new file mode 100644 index 00000000..f097cc9b --- /dev/null +++ b/components/adapters/pi/src/pi-types.ts @@ -0,0 +1,142 @@ +/** + * Minimal structural mirrors of the pi extension API this adapter touches. + * + * Following the DeepSeek Harness adapter's precedent (`deepseek-harness/src/types.ts`), + * the adapter does not import pi's packages: pi loads it in-process and passes + * real objects that match these shapes. Every field is mirrored from the pinned + * `@earendil-works/pi-coding-agent@0.87.1` declarations: + * - ExtensionAPI / events / results dist/core/extensions/types.d.ts + * - SessionEntry / ContextEditEntry dist/core/session-manager.d.ts + * - AgentMessage content types @earendil-works/pi-ai dist/types.d.ts + * - custom message roles dist/core/messages.d.ts + * If pi changes one of these shapes, this file is the single place to update. + */ + +export type PiTextContent = { type: "text"; text: string; textSignature?: string }; +export type PiThinkingContent = { type: "thinking"; thinking: string; thinkingSignature?: string; redacted?: boolean }; +export type PiImageContent = { type: "image"; data: string; mimeType: string }; +export type PiToolCall = { type: "toolCall"; id: string; name: string; arguments: Record }; + +export type PiUserMessage = { role: "user"; content: string | (PiTextContent | PiImageContent)[]; timestamp: number }; +export type PiAssistantMessage = { + role: "assistant"; + content: (PiTextContent | PiThinkingContent | PiToolCall)[]; + model?: string; + usage?: { input: number; output: number; cacheRead: number; cacheWrite: number }; + timestamp: number; + [key: string]: unknown; +}; +export type PiToolResultMessage = { + role: "toolResult"; + toolCallId: string; + toolName: string; + content: (PiTextContent | PiImageContent)[]; + isError: boolean; + timestamp: number; + [key: string]: unknown; +}; +/** bashExecution, custom, branchSummary, compactionSummary, and future roles. */ +export type PiOtherMessage = { role: string; [key: string]: unknown }; + +export type PiAgentMessage = PiUserMessage | PiAssistantMessage | PiToolResultMessage | PiOtherMessage; + +export type PiSessionEntry = { + type: string; + id: string; + parentId: string | null; + timestamp: string; + message?: PiAgentMessage; + targetId?: string; + customType?: string; + [key: string]: unknown; +}; + +export type PiProjectedSessionEntry = { + sourceEntry: PiSessionEntry; + messages: PiAgentMessage[]; +}; + +export type PiReadonlySessionManager = { + getCwd(): string; + getSessionId(): string; + getSessionFile(): string | undefined; + getBranch(fromId?: string): PiSessionEntry[]; +}; + +export type PiExtensionContext = { + cwd: string; + mode?: string; + sessionManager: PiReadonlySessionManager; + model?: { id?: string; provider?: string } | undefined; + getSystemPrompt?(): string; +}; + +export type PiContextEditDraft = { + type: "context_edit"; + targetId: string; + replacement: { content: string | (PiTextContent | PiImageContent)[] } | null; +}; + +export type PiBoundaryState = { + entries: unknown[]; + continue: boolean; + context: { contextEntries: PiProjectedSessionEntry[] }; +}; + +export type PiEventMap = { + session_start: { type: "session_start"; reason: string }; + session_shutdown: { type: "session_shutdown"; reason: string }; + before_agent_start: { + type: "before_agent_start"; + prompt: string; + readonly systemPrompt: string; + systemPromptOptions: { + appendSystemPrompt: string; + sections: Record; + contextFiles: Array<{ path: string; content: string }>; + [key: string]: unknown; + }; + }; + context: { type: "context"; messages: PiAgentMessage[] }; + turn_start: { type: "turn_start"; turnIndex: number; timestamp: number }; + turn_end: PiBoundaryState & { type: "turn_end"; turnIndex: number }; + message_end: { type: "message_end"; message: PiAgentMessage }; +}; + +export type PiEventResultMap = { + session_start: void; + session_shutdown: void; + before_agent_start: { message?: { customType: string; content: string; display: boolean }; systemPrompt?: string } | void; + context: { messages?: PiAgentMessage[] } | void; + turn_start: void; + turn_end: { entries?: PiContextEditDraft[]; continue?: boolean } | void; + message_end: void; +}; + +export type PiToolResult = { + content: PiTextContent[]; + details: unknown; +}; + +export type PiToolDefinition = { + name: string; + label: string; + description: string; + promptSnippet?: string; + parameters: Record; + execute( + toolCallId: string, + params: Record, + signal: AbortSignal | undefined, + onUpdate: unknown, + ctx: PiExtensionContext, + ): Promise; +}; + +export type PiExtensionAPI = { + on( + event: K, + handler: (event: PiEventMap[K], ctx: PiExtensionContext) => Promise | PiEventResultMap[K], + ): unknown; + registerTool(tool: PiToolDefinition): void; +}; diff --git a/components/adapters/pi/src/preset.ts b/components/adapters/pi/src/preset.ts new file mode 100644 index 00000000..7250091b --- /dev/null +++ b/components/adapters/pi/src/preset.ts @@ -0,0 +1,17 @@ +import { + createTokenPilotHostBinding, + initializeTokenPilotPreset, +} from "@lightrsi/tokenpilot"; + +export const PI_TOKENPILOT_HOST_BINDING = createTokenPilotHostBinding({ + hostId: "pi", + supportedFeatures: ["stabilizer", "reduction", "eviction"], +}); + +export function initializePiTokenPilotPreset(): void { + initializeTokenPilotPreset(PI_TOKENPILOT_HOST_BINDING); +} + +export function piSupportsEviction(): boolean { + return PI_TOKENPILOT_HOST_BINDING.supportedFeatures.includes("eviction"); +} diff --git a/components/adapters/pi/src/product-registration.ts b/components/adapters/pi/src/product-registration.ts new file mode 100644 index 00000000..b8588a93 --- /dev/null +++ b/components/adapters/pi/src/product-registration.ts @@ -0,0 +1,17 @@ +import { readLatestUxEffect } from "@lightrsi/host-adapter"; +import { defineProductHostRegistration } from "@lightrsi/product-surface"; +import { defaultTokenPilotPiConfigPath, loadTokenPilotPiConfig } from "./config.js"; +import { PI_TOKENPILOT_HOST_BINDING } from "./preset.js"; + +export const PI_PRODUCT_HOST_REGISTRATION = defineProductHostRegistration({ + hostId: "pi", + displayName: "pi", + preset: PI_TOKENPILOT_HOST_BINDING, + async resolveStateDir(context) { + const config = await loadTokenPilotPiConfig( + context?.productConfigPath?.trim() || defaultTokenPilotPiConfigPath(), + ); + return config.stateDir; + }, + readLatestActivity: readLatestUxEffect, +}); diff --git a/components/adapters/pi/src/runtime.ts b/components/adapters/pi/src/runtime.ts new file mode 100644 index 00000000..903a64d9 --- /dev/null +++ b/components/adapters/pi/src/runtime.ts @@ -0,0 +1,337 @@ +/** + * TokenPilot runtime for pi. Each pi hook maps to one shared TokenPilot path: + * + * session_start load config, bind state roots, reset per-session caches + * before_agent_start stable prefix (shared stabilizer over prompt sections) and + * the shared recovery protocol as a stable prompt section + * context request-time reduction (shared before-call pipeline) + * turn_end opt-in lifecycle eviction → native `context_edit` entries + * (committed before pi's threshold compaction check) + * memory_fault_recover native tool over the shared recovery resolver + * + * Every hook fails open: on any error pi continues with its own, unmodified data + * and the reason is logged to `/tokenpilot/adapter.log`. + */ +import { configureStatePathResolver } from "@lightrsi/artifact-store"; +import type { TaskStateEstimator } from "@lightrsi/eviction"; +import type { SessionTaskRegistry } from "@lightrsi/history"; +import { + appendRecentTurnBinding, + createStaticStatePathResolver, + writeLatestSessionRef, +} from "@lightrsi/host-adapter"; +import type { RuntimeMessage } from "@lightrsi/kernel"; +import { MEMORY_FAULT_RECOVER_TOOL_NAME, resolveMemoryFaultRecover } from "@lightrsi/mcp"; +import { prependTextToContent, rewriteTextForStablePrefix } from "@lightrsi/stabilizer"; +import { + canonicalRecoveryProtocolText, + createCanonicalEnvelope, + recordCanonicalStability, + runCanonicalBeforeCallReduction, +} from "../../shared/canonical/before-call.js"; +import { evaluateEvictionReadiness } from "../../shared/canonical/config.js"; +import { + createCanonicalEstimator, + runCanonicalSurfaceEviction, + type CanonicalSurfaceEntry, +} from "../../shared/canonical/eviction.js"; +import { createFileLogger, failOpen, type AdapterLogger } from "../../shared/canonical/logger.js"; +import { loadReductionMemo, saveReductionMemoIfDirty } from "../../shared/canonical/memo-store.js"; +import { ReductionMemo, passSavedChars } from "../../shared/canonical/reduction.js"; +import { + decodePiMessages, + encodePiMessages, + piProjectionToSurfaceEntries, + piReplacementContent, +} from "./codec.js"; +import { loadTokenPilotPiConfig, type TokenPilotPiConfig } from "./config.js"; +import type { + PiAgentMessage, + PiContextEditDraft, + PiEventMap, + PiEventResultMap, + PiExtensionContext, + PiToolDefinition, +} from "./pi-types.js"; +import { initializePiTokenPilotPreset, piSupportsEviction } from "./preset.js"; + +export const PI_HOST_ID = "pi"; +export const PI_DISPLAY_NAME = "pi"; +/** Trailing section that carries volatile prompt lines (developer target). */ +export const PI_DYNAMIC_SECTION = "tokenpilot_dynamic"; +/** Stable section that carries the shared recovery protocol. */ +export const PI_RECOVERY_SECTION = "tokenpilot_recovery"; + +export type PiRuntimeDependencies = { + loadConfig?: () => Promise; + supportsEviction?: boolean; + createEstimator?: (config: TokenPilotPiConfig) => TaskStateEstimator | undefined; + registryStore?: { + load(sessionId: string): Promise | SessionTaskRegistry; + persist(registry: SessionTaskRegistry, expectedVersion: number): Promise | void; + }; +}; + +const RECOVERY_TOOL_SCHEMA = { + type: "object", + additionalProperties: false, + properties: { + dataKey: { type: "string", description: "Archive dataKey from a prior [Tool payload trimmed] notice." }, + startLine: { type: "integer", minimum: 1, description: "Optional 1-based start line for partial recovery." }, + endLine: { type: "integer", minimum: 1, description: "Optional 1-based end line for partial recovery." }, + }, + required: ["dataKey"], +} as const; + +export class PiTokenPilotRuntime { + config: TokenPilotPiConfig | undefined; + sessionId: string | undefined; + /** Disclosed-read memo of `memoSessionId`, persisted under stateDir so pi restarts still honour deliberate re-reads. */ + memo = new ReductionMemo(); + private memoSessionId: string | undefined; + readonly logger: AdapterLogger; + /** Volatile prompt lines to prepend request-locally to the first user message (user target). */ + pendingUserPrefix: string | undefined; + private readonly deps: Required> + & Pick; + + constructor(deps: PiRuntimeDependencies = {}) { + this.deps = { + loadConfig: deps.loadConfig ?? (() => loadTokenPilotPiConfig()), + supportsEviction: deps.supportsEviction ?? piSupportsEviction(), + createEstimator: deps.createEstimator ?? createCanonicalEstimator, + registryStore: deps.registryStore, + }; + this.logger = createFileLogger({ + hostId: PI_HOST_ID, + stateDir: () => this.config?.stateDir, + debug: () => this.config?.logLevel === "debug", + }); + } + + private active(): TokenPilotPiConfig | undefined { + return this.config?.enabled ? this.config : undefined; + } + + private resolveSessionId(ctx: PiExtensionContext): string { + const id = ctx.sessionManager?.getSessionId?.(); + return typeof id === "string" && id.trim() ? id.trim() : this.sessionId ?? "pi-unknown-session"; + } + + onSessionStart(_event: PiEventMap["session_start"], ctx: PiExtensionContext): Promise { + return failOpen(this.logger, "session_start", async () => { + this.memo.clear(); + this.pendingUserPrefix = undefined; + this.config = await this.deps.loadConfig(); + const config = this.config; + initializePiTokenPilotPreset(); + configureStatePathResolver(createStaticStatePathResolver({ + hostId: PI_HOST_ID, + displayName: PI_DISPLAY_NAME, + stateDir: config.stateDir, + namespaceDir: "tokenpilot", + })); + this.sessionId = this.resolveSessionId(ctx); + this.memoSessionId = undefined; + if (!config.enabled) return; + await this.memoFor(config, this.sessionId); + await writeLatestSessionRef(config.stateDir, this.sessionId, new Date().toISOString()); + this.logger.info("session_start", { sessionId: this.sessionId, cwd: ctx.cwd }); + }, undefined); + } + + private async memoFor(config: TokenPilotPiConfig, sessionId: string): Promise { + if (this.memoSessionId !== sessionId) { + this.memo = await loadReductionMemo(config.stateDir, sessionId); + this.memoSessionId = sessionId; + } + return this.memo; + } + + onSessionShutdown(): void { + this.memo.clear(); + this.memoSessionId = undefined; + this.pendingUserPrefix = undefined; + } + + onBeforeAgentStart( + event: PiEventMap["before_agent_start"], + ctx: PiExtensionContext, + ): Promise { + return failOpen(this.logger, "before_agent_start", async () => { + const config = this.active(); + if (!config) return undefined; + const options = event.systemPromptOptions; + const before = event.systemPrompt; + const dynamicLines: string[] = []; + + if (config.modules.stabilizer) { + const take = (text: string): string => { + const rewrite = rewriteTextForStablePrefix(text); + if (!rewrite.changed || !rewrite.dynamicContextText) return text; + for (const line of rewrite.dynamicContextText.split("\n")) { + if (line.trim() && !dynamicLines.includes(line)) dynamicLines.push(line); + } + return rewrite.forwardedText; + }; + if (typeof options.appendSystemPrompt === "string" && options.appendSystemPrompt) { + options.appendSystemPrompt = take(options.appendSystemPrompt); + } + for (const [name, text] of Object.entries(options.sections ?? {})) { + if (name === PI_DYNAMIC_SECTION || name === PI_RECOVERY_SECTION || typeof text !== "string") continue; + options.sections[name] = take(text); + } + for (const file of Array.isArray(options.contextFiles) ? options.contextFiles : []) { + if (file && typeof file.content === "string") file.content = take(file.content); + } + } + + const dynamicText = dynamicLines.join("\n"); + this.pendingUserPrefix = undefined; + if (dynamicText) { + if (config.hooks.dynamicContextTarget === "developer") options.sections[PI_DYNAMIC_SECTION] = dynamicText; + else this.pendingUserPrefix = dynamicText; + } + const recovery = canonicalRecoveryProtocolText(); + if (recovery) options.sections[PI_RECOVERY_SECTION] = recovery; + + if (dynamicText) { + await recordCanonicalStability({ + config, + sessionId: this.resolveSessionId(ctx), + model: ctx.model?.id ?? "", + before, + result: { + changed: true, + instructions: typeof ctx.getSystemPrompt === "function" ? ctx.getSystemPrompt() : before, + dynamicContextText: dynamicText, + }, + }).catch((error) => this.logger.warn("stability snapshot failed", error)); + } + return undefined; + }, undefined); + } + + onContext(event: PiEventMap["context"], ctx: PiExtensionContext): Promise { + return failOpen(this.logger, "context", async () => { + const config = this.active(); + if (!config || !Array.isArray(event.messages)) return undefined; + const original = event.messages; + let canonical: RuntimeMessage[] = decodePiMessages(original); + + if (this.pendingUserPrefix) { + const userIndex = canonical.findIndex((message) => message.role === "user" && message.metadata?.lightrsiPassThrough !== true); + if (userIndex >= 0) { + const target = canonical[userIndex]!; + canonical = canonical.slice(); + canonical[userIndex] = { ...target, content: prependTextToContent(target.content, this.pendingUserPrefix) as RuntimeMessage["content"] }; + } + } + + const sessionId = this.resolveSessionId(ctx); + if (config.modules.reduction) { + const reduced = await runCanonicalBeforeCallReduction({ + envelope: createCanonicalEnvelope({ + hostId: PI_HOST_ID, + displayName: PI_DISPLAY_NAME, + sessionId, + model: ctx.model?.id ?? "", + messages: canonical, + }), + config, + memo: await this.memoFor(config, sessionId), + }); + canonical = reduced.messages; + await saveReductionMemoIfDirty(config.stateDir, sessionId, this.memo) + .catch((error) => this.logger.warn("reduction memo save failed", error)); + if ((reduced.summary?.savedChars ?? 0) > 0) { + await appendRecentTurnBinding(config.stateDir, { + sessionId, + updatedAt: new Date().toISOString(), + model: ctx.model?.id, + reductionSavedChars: reduced.summary?.savedChars, + reductionChangedBlocks: reduced.summary?.changedBlocks, + reductionPassSavedChars: passSavedChars(reduced.summary), + }).catch((error) => this.logger.warn("turn binding failed", error)); + } + } + + const encoded = encodePiMessages(original, canonical); + return encoded.some((message, index) => message !== original[index]) ? { messages: encoded } : undefined; + }, undefined); + } + + onTurnEnd(event: PiEventMap["turn_end"], ctx: PiExtensionContext): Promise { + return failOpen(this.logger, "turn_end", async () => { + const config = this.active(); + if (!config) return undefined; + const readiness = evaluateEvictionReadiness(config, this.deps.supportsEviction); + if (!readiness.active) { + this.logger.debug("eviction inactive", readiness); + return undefined; + } + const estimator = this.deps.createEstimator(config); + if (!estimator) return undefined; + const projected = Array.isArray(event.context?.contextEntries) ? event.context.contextEntries : []; + const entries: CanonicalSurfaceEntry[] = piProjectionToSurfaceEntries(projected); + if (entries.length === 0) return undefined; + const originals = new Map( + projected.map((entry) => [entry?.sourceEntry?.id, entry?.messages?.[0]]), + ); + const drafts: PiContextEditDraft[] = []; + const sessionId = this.resolveSessionId(ctx); + const cycle = await runCanonicalSurfaceEviction({ + sessionId, + entries, + config, + estimator, + registryStore: this.deps.registryStore, + apply(replacements) { + for (const replacement of replacements) { + drafts.push({ + type: "context_edit", + targetId: replacement.id, + replacement: { content: piReplacementContent(originals.get(replacement.id), replacement.text) }, + }); + } + return "committed"; + }, + }); + if (cycle.status !== "applied" || drafts.length === 0) return undefined; + this.logger.info("eviction applied", { + sessionId, + targets: drafts.map((draft) => draft.targetId), + savedChars: cycle.result.replacements.reduce((sum, r) => sum + Math.max(0, r.originalChars - r.text.length), 0), + }); + return { entries: drafts }; + }, undefined); + } + + recoveryTool(): PiToolDefinition { + return { + name: MEMORY_FAULT_RECOVER_TOOL_NAME, + label: "TokenPilot recovery", + description: + "Recover archived content that was trimmed from a prior tool result. Use this internal tool with the provided dataKey instead of re-running the original tool.", + parameters: RECOVERY_TOOL_SCHEMA as unknown as Record, + execute: async (_toolCallId, params) => { + const dataKey = typeof params?.dataKey === "string" ? params.dataKey : ""; + try { + const result = await resolveMemoryFaultRecover({ + dataKey, + stateDir: this.config?.stateDir, + startLine: typeof params?.startLine === "number" ? params.startLine : undefined, + endLine: typeof params?.endLine === "number" ? params.endLine : undefined, + }); + return { content: [{ type: "text", text: result.text }], details: result.details }; + } catch (error) { + this.logger.warn("memory_fault_recover failed", error); + return { + content: [{ type: "text", text: `Recovery failed for dataKey: ${dataKey}` }], + details: { error: "recovery_failed", dataKey }, + }; + } + }, + }; + } +} diff --git a/components/adapters/pi/tests/binding-registration.test.ts b/components/adapters/pi/tests/binding-registration.test.ts new file mode 100644 index 00000000..89c3492c --- /dev/null +++ b/components/adapters/pi/tests/binding-registration.test.ts @@ -0,0 +1,58 @@ +/** + * Regression matrix: pi/src/preset.ts + pi/src/product-registration.ts + config paths + * + * B1 host binding: hostId "pi", TokenPilot preset id/version, features exactly + * [stabilizer, reduction, eviction]; initializing the preset does not throw + * B2 product registration: hostId, display name, binding equal to the declared one + * B3 resolveStateDir honours an explicit product config path, else the default + * P1 default paths derive from PI_CODING_AGENT_DIR; TOKENPILOT_PI_CONFIG overrides the config + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { TOKENPILOT_PRESET_ID, TOKENPILOT_PRESET_VERSION } from "@lightrsi/tokenpilot"; + +import { defaultPiExtensionDir, defaultPiStateDir, defaultTokenPilotPiConfigPath } from "../src/config.js"; +import { PI_TOKENPILOT_HOST_BINDING, initializePiTokenPilotPreset, piSupportsEviction } from "../src/preset.js"; +import { PI_PRODUCT_HOST_REGISTRATION } from "../src/product-registration.js"; + +describe("pi binding and registration", () => { + it("B1 host binding declares the supported subset", () => { + assert.equal(PI_TOKENPILOT_HOST_BINDING.hostId, "pi"); + assert.equal(PI_TOKENPILOT_HOST_BINDING.presetId, TOKENPILOT_PRESET_ID); + assert.equal(PI_TOKENPILOT_HOST_BINDING.presetVersion, TOKENPILOT_PRESET_VERSION); + assert.deepEqual(PI_TOKENPILOT_HOST_BINDING.supportedFeatures, ["stabilizer", "reduction", "eviction"]); + assert.equal(piSupportsEviction(), true); + assert.doesNotThrow(() => initializePiTokenPilotPreset()); + }); + it("B2 product registration", () => { + assert.equal(PI_PRODUCT_HOST_REGISTRATION.hostId, "pi"); + assert.equal(PI_PRODUCT_HOST_REGISTRATION.displayName, "pi"); + assert.deepEqual(PI_PRODUCT_HOST_REGISTRATION.preset, PI_TOKENPILOT_HOST_BINDING); + }); + it("B3 resolveStateDir uses the product config path", async () => { + const dir = await mkdtemp(join(tmpdir(), "tp-pi-reg-")); + const configPath = join(dir, "tokenpilot.json"); + await writeFile(configPath, JSON.stringify({ stateDir: join(dir, "custom-state") })); + assert.equal(await PI_PRODUCT_HOST_REGISTRATION.resolveStateDir({ productConfigPath: configPath }), join(dir, "custom-state")); + }); + it("P1 default paths follow PI_CODING_AGENT_DIR and TOKENPILOT_PI_CONFIG", () => { + const saved = { agent: process.env.PI_CODING_AGENT_DIR, cfg: process.env.TOKENPILOT_PI_CONFIG }; + try { + process.env.PI_CODING_AGENT_DIR = "/agents/pi"; + delete process.env.TOKENPILOT_PI_CONFIG; + assert.equal(defaultTokenPilotPiConfigPath(), "/agents/pi/tokenpilot.json"); + assert.equal(defaultPiStateDir(), "/agents/pi/tokenpilot-state/tokenpilot"); + assert.equal(defaultPiExtensionDir(), "/agents/pi/extensions/tokenpilot"); + process.env.TOKENPILOT_PI_CONFIG = "/cfg/tp.json"; + assert.equal(defaultTokenPilotPiConfigPath(), "/cfg/tp.json"); + assert.equal(defaultPiStateDir(), "/cfg/tokenpilot-state/tokenpilot"); + } finally { + if (saved.agent === undefined) delete process.env.PI_CODING_AGENT_DIR; else process.env.PI_CODING_AGENT_DIR = saved.agent; + if (saved.cfg === undefined) delete process.env.TOKENPILOT_PI_CONFIG; else process.env.TOKENPILOT_PI_CONFIG = saved.cfg; + } + }); +}); diff --git a/components/adapters/pi/tests/codec.test.ts b/components/adapters/pi/tests/codec.test.ts new file mode 100644 index 00000000..16b71f6a --- /dev/null +++ b/components/adapters/pi/tests/codec.test.ts @@ -0,0 +1,212 @@ +/** + * Regression matrix: pi/src/codec.ts (pi AgentMessage ↔ canonical transcript bridge) + * + * decodePiMessage — one case per pi role + * D1 user string content → user, same string + * D2 user parts (text + image) → user text/image blocks (image keeps mimeType only) + * D3 assistant text/toolCall/thinking → text + tool_call blocks; thinking in metadata.reasoning + * D4 toolResult (text + image, error) → tool role, one tool_result (joined text, error status, + * imageParts count) + * D5 bashExecution / custom / branchSummary / compactionSummary / unknown → user pass-through + * with their visible text + * D6 malformed (non-object, missing content) → pass-through, never throws + * D7 metadata carries piIndex and piRole + * encodePiMessages + * E1 round trip: encode(original, decode(original)) returns the SAME objects + * E2 changed tool-result text → cloned toolResult with new text first, images kept, other + * fields identical; original untouched + * E3 changed user string content → cloned user with new string + * E4 changed user parts → cloned user with new text part first, images kept + * E5 changes on assistant / pass-through messages are ignored (not writable) + * E6 a shorter canonical array leaves the remaining originals untouched + * piProjectionToSurfaceEntries (projected branch → surface entries) + * P1 message entries become items with the entry id; turn increments on each user message + * P2 compaction / branch_summary / custom_message entries become checkpoints (`#`) + * P3 system messages and empty projections are skipped + * P4 an entry already replaced by an eviction stub is flagged alreadyEvicted + * P5 pass-through roles inside message entries (bashExecution) are checkpoints + * piReplacementContent + * R1 user with string content → string; R2 everything else → single text part + * Round-trip fixtures cover tool calls/results, reasoning, images, branch and compaction entries. + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; + +import { + decodePiMessage, + decodePiMessages, + encodePiMessages, + piProjectionToSurfaceEntries, + piReplacementContent, +} from "../src/codec.js"; +import type { PiAgentMessage, PiProjectedSessionEntry } from "../src/pi-types.js"; + +const IMAGE = { type: "image", data: "iVBORw0KGgo=", mimeType: "image/png" } as const; + +function transcript(): PiAgentMessage[] { + return [ + { role: "user", content: "fix the build", timestamp: 1 }, + { role: "user", content: [{ type: "text", text: "see screenshot" }, IMAGE], timestamp: 2 }, + { + role: "assistant", + content: [ + { type: "thinking", thinking: "plan: run make" }, + { type: "text", text: "Running make." }, + { type: "toolCall", id: "call_1", name: "bash", arguments: { command: "make" } }, + ], + model: "qwen", + timestamp: 3, + }, + { role: "toolResult", toolCallId: "call_1", toolName: "bash", content: [{ type: "text", text: "ok\n" }, { type: "text", text: "done" }, IMAGE], isError: true, timestamp: 4 }, + { role: "bashExecution", command: "ls", output: "a b", exitCode: 0, cancelled: false, truncated: false, timestamp: 5 }, + { role: "custom", customType: "note", content: "remember this", display: true, timestamp: 6 }, + { role: "branchSummary", summary: "tried X", fromId: null, timestamp: 7 }, + { role: "compactionSummary", summary: "earlier work", tokensBefore: 900, timestamp: 8 }, + ]; +} + +describe("decodePiMessage", () => { + const decoded = decodePiMessages(transcript()); + it("D1 user string content", () => { + assert.deepEqual(decoded[0], { role: "user", content: "fix the build", metadata: { piIndex: 0, piRole: "user" } }); + }); + it("D2 user text and image parts", () => { + assert.deepEqual(decoded[1]?.content, [{ type: "text", text: "see screenshot" }, { type: "image", mediaType: "image/png" }]); + }); + it("D3 assistant text, tool call and reasoning", () => { + assert.deepEqual(decoded[2], { + role: "assistant", + content: [ + { type: "text", text: "Running make." }, + { type: "tool_call", toolCallId: "call_1", toolName: "bash", argumentsJson: { command: "make" } }, + ], + metadata: { piIndex: 2, piRole: "assistant", reasoning: ["plan: run make"] }, + }); + }); + it("D4 tool result with text, error status and images", () => { + assert.deepEqual(decoded[3], { + role: "tool", + content: [{ type: "tool_result", toolCallId: "call_1", toolName: "bash", status: "error", text: "ok\n\ndone" }], + metadata: { piIndex: 3, piRole: "toolResult", imageParts: 1 }, + }); + }); + it("D5 pi-specific roles are user pass-through with visible text", () => { + assert.deepEqual(decoded.slice(4).map((m) => [m.role, m.content, m.metadata?.lightrsiPassThrough]), [ + ["user", "$ ls\na b", true], + ["user", "remember this", true], + ["user", "tried X", true], + ["user", "earlier work", true], + ]); + assert.equal(decodePiMessage({ role: "futureRole", content: [{ type: "text", text: "t" }] }, 0).content, "t"); + }); + it("D6 malformed input never throws", () => { + assert.doesNotThrow(() => decodePiMessage(null as unknown as PiAgentMessage, 0)); + assert.equal(decodePiMessage(null as unknown as PiAgentMessage, 0).metadata?.lightrsiPassThrough, true); + assert.deepEqual(decodePiMessage({ role: "assistant" } as PiAgentMessage, 0).content, []); + assert.deepEqual(decodePiMessage({ role: "toolResult" } as PiAgentMessage, 0).content, [{ type: "tool_result", toolCallId: undefined, toolName: undefined, status: "success", text: "" }]); + }); + it("D7 metadata carries index and role", () => { + assert.deepEqual(decoded.map((m) => m.metadata?.piIndex), [0, 1, 2, 3, 4, 5, 6, 7]); + assert.equal(decoded[6]?.metadata?.piRole, "branchSummary"); + }); +}); + +describe("encodePiMessages", () => { + it("E1 an unchanged round trip returns the same objects", () => { + const original = transcript(); + const encoded = encodePiMessages(original, decodePiMessages(original)); + assert.equal(encoded.length, original.length); + encoded.forEach((message, index) => assert.equal(message, original[index], `index ${index}`)); + }); + it("E2 changed tool-result text clones the message and keeps images", () => { + const original = transcript(); + const snapshot = JSON.stringify(original); + const canonical = decodePiMessages(original); + canonical[3] = { ...canonical[3]!, content: [{ type: "tool_result", toolCallId: "call_1", toolName: "bash", status: "error", text: "trimmed" }] }; + const encoded = encodePiMessages(original, canonical); + assert.notEqual(encoded[3], original[3]); + assert.deepEqual(encoded[3], { ...original[3], content: [{ type: "text", text: "trimmed" }, IMAGE] }); + assert.equal(JSON.stringify(original), snapshot, "original mutated"); + }); + it("E3 changed user string content", () => { + const original = transcript(); + const canonical = decodePiMessages(original); + canonical[0] = { ...canonical[0]!, content: "DATE\n\nfix the build" }; + assert.deepEqual(encodePiMessages(original, canonical)[0], { role: "user", content: "DATE\n\nfix the build", timestamp: 1 }); + }); + it("E4 changed user parts keep images after the new text", () => { + const original = transcript(); + const canonical = decodePiMessages(original); + canonical[1] = { ...canonical[1]!, content: [{ type: "text", text: "new" }, { type: "image", mediaType: "image/png" }] }; + assert.deepEqual(encodePiMessages(original, canonical)[1], { role: "user", content: [{ type: "text", text: "new" }, IMAGE], timestamp: 2 }); + }); + it("E5 assistant and pass-through changes are ignored", () => { + const original = transcript(); + const canonical = decodePiMessages(original); + canonical[2] = { ...canonical[2]!, content: [{ type: "text", text: "rewritten" }] }; + canonical[5] = { ...canonical[5]!, content: "rewritten" }; + const encoded = encodePiMessages(original, canonical); + assert.equal(encoded[2], original[2]); + assert.equal(encoded[5], original[5]); + }); + it("E6 a shorter canonical array leaves remaining messages untouched", () => { + const original = transcript(); + const encoded = encodePiMessages(original, decodePiMessages(original).slice(0, 2)); + encoded.forEach((message, index) => assert.equal(message, original[index])); + }); +}); + +describe("piProjectionToSurfaceEntries", () => { + const entry = (id: string, type: string, messages: PiAgentMessage[]): PiProjectedSessionEntry => ({ + sourceEntry: { type, id, parentId: null, timestamp: "t" }, + messages, + }); + function projection(): PiProjectedSessionEntry[] { + const t = transcript(); + return [ + entry("cmp", "compaction", [{ role: "system", content: "checkpoint" }, t[7]!]), + entry("e1", "message", [t[0]!]), + entry("e2", "message", [t[2]!]), + entry("e3", "message", [t[3]!]), + entry("bs", "branch_summary", [t[6]!]), + entry("cm", "custom_message", [t[5]!]), + entry("bx", "message", [t[4]!]), + entry("st", "custom", []), + entry("e4", "message", [{ role: "user", content: "next", timestamp: 9 }]), + entry("e5", "message", [{ role: "toolResult", toolCallId: "c2", toolName: "bash", content: [{ type: "text", text: "[evicted: tool_result e5]\n\nhint" }], isError: false, timestamp: 10 }]), + ]; + } + const entries = () => piProjectionToSurfaceEntries(projection()); + + it("P1 message entries keep their id and turns start at user messages", () => { + assert.deepEqual(entries().filter((e) => !e.checkpoint).map((e) => [e.id, e.turn]), [["e1", 1], ["e2", 1], ["e3", 1], ["e4", 2], ["e5", 2]]); + }); + it("P2 compaction, branch summary and custom messages are checkpoints", () => { + assert.deepEqual(entries().filter((e) => e.checkpoint).map((e) => e.id), ["cmp#1", "bs#0", "cm#0", "bx"]); + }); + it("P3 system messages and empty projections are skipped", () => { + const ids = entries().map((e) => e.id); + assert.ok(!ids.includes("cmp#0")); + assert.ok(!ids.some((id) => id.startsWith("st"))); + }); + it("P4 eviction stubs are flagged alreadyEvicted", () => { + assert.deepEqual(entries().filter((e) => e.alreadyEvicted).map((e) => e.id), ["e5"]); + }); + it("P5 bashExecution inside a message entry is a checkpoint", () => { + assert.equal(entries().find((e) => e.id === "bx")?.checkpoint, true); + }); + it("malformed projections are skipped, not thrown", () => { + assert.deepEqual(piProjectionToSurfaceEntries([null as unknown as PiProjectedSessionEntry, { sourceEntry: { type: "message" } } as unknown as PiProjectedSessionEntry]), []); + }); +}); + +describe("piReplacementContent", () => { + it("R1 a user message with string content gets a string", () => { + assert.equal(piReplacementContent({ role: "user", content: "x", timestamp: 1 }, "stub"), "stub"); + }); + it("R2 everything else gets a single text part", () => { + assert.deepEqual(piReplacementContent({ role: "toolResult", toolCallId: "c", toolName: "b", content: [], isError: false, timestamp: 1 }, "stub"), [{ type: "text", text: "stub" }]); + assert.deepEqual(piReplacementContent({ role: "user", content: [{ type: "text", text: "x" }], timestamp: 1 }, "stub"), [{ type: "text", text: "stub" }]); + assert.deepEqual(piReplacementContent(undefined, "stub"), [{ type: "text", text: "stub" }]); + }); +}); diff --git a/components/adapters/pi/tests/install-doctor.test.ts b/components/adapters/pi/tests/install-doctor.test.ts new file mode 100644 index 00000000..a99cf117 --- /dev/null +++ b/components/adapters/pi/tests/install-doctor.test.ts @@ -0,0 +1,286 @@ +/** + * Regression matrix: pi/src/install.ts + pi/src/doctor.ts + * + * installPiTokenPilot + * I1 missing bundle → throws, nothing written + * I2 fresh agent dir → marker loader requiring the bundle; normal-mode config created; + * manifest (configCreated) and state dir written; CLI context remembers pi's config + * I3 existing TokenPilot loader → overwritten in place, no backup + * I4 foreign file at the loader path → moved to a timestamped backup, recorded in manifest + * I5 existing config → kept byte-identical (user settings and credentials untouched) + * I6 re-install keeps the first install's backup path and configCreated flag + * I7 the created config carries no credentials + * renderPiLoader / isTokenPilotPiLoader / loaderTarget + * L1 the loader's module value is the bundle's default factory, with .default === itself + * (pi's jiti import requires a function; regression for a failed live load) + * L2 marker detection: ours → true; foreign → false; missing → false + * L3 loaderTarget parses the required path; garbage → undefined + * uninstallPiTokenPilot + * U1 removes our loader; keeps config and state without --purge + * U2 a foreign loader is left untouched + * U3 the recorded backup is restored + * U4 --purge removes a config it created and the state dir + * U5 --purge keeps a config that existed before install + * U6 uninstall without an install is a no-op + * U7 install → uninstall(--purge) leaves no TokenPilot files in the agent dir + * inspectPiDoctor / formatPiDoctorReport + * D1 fresh install → healthy + * D2 nothing installed → unhealthy with config and loader problems + * D3 loader points to a missing bundle → problem + * D4 disabled config → problem + * D5 eviction requested but estimator incomplete → problem naming missing keys + * D6 latest session and activity are reported once written + * D7 formatted report lists every check and the health verdict + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync } from "node:fs"; +import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { recordUxEffect, writeLatestSessionRef } from "@lightrsi/host-adapter"; + +import { loadTokenPilotPiConfig, normalizeTokenPilotPiConfig } from "../src/config.js"; +import { formatPiDoctorReport, inspectPiDoctor, loaderTarget } from "../src/doctor.js"; +import { + PI_LOADER_MARKER, + installPiTokenPilot, + isTokenPilotPiLoader, + renderPiLoader, + uninstallPiTokenPilot, +} from "../src/install.js"; + +async function fixture() { + const root = await mkdtemp(join(tmpdir(), "tp-pi-install-")); + const agentDir = join(root, "agent"); + const bundlePath = join(root, "bundle", "extension.js"); + await mkdir(join(root, "bundle"), { recursive: true }); + await writeFile(bundlePath, "module.exports = { default: function tokenpilot() { return 'ok'; }, marker: 42 };\n"); + const cliContextPath = join(root, "cli-context.json"); + const install = (extra: Record = {}) => installPiTokenPilot({ + agentDir, bundlePath, cliContextPath, installCliBin: false, ...extra, + }); + return { root, agentDir, bundlePath, cliContextPath, install, loaderPath: join(agentDir, "extensions", "tokenpilot", "index.js"), configPath: join(agentDir, "tokenpilot.json") }; +} + +describe("installPiTokenPilot", () => { + it("I1 a missing bundle throws before writing anything", async () => { + const f = await fixture(); + await assert.rejects(f.install({ bundlePath: join(f.root, "none.js") }), /not built/); + assert.equal(existsSync(f.agentDir), false); + }); + it("I2 a fresh install writes loader, config, manifest, state dir and CLI context", async () => { + const f = await fixture(); + const result = await f.install(); + assert.equal(await readFile(f.loaderPath, "utf8"), renderPiLoader(f.bundlePath)); + assert.equal(result.configCreated, true); + assert.deepEqual(await loadTokenPilotPiConfig(f.configPath), normalizeTokenPilotPiConfig({}, { configPath: f.configPath })); + const manifest = JSON.parse(await readFile(join(f.agentDir, "tokenpilot-install.json"), "utf8")); + assert.equal(manifest.configCreated, true); + assert.equal(manifest.bundlePath, f.bundlePath); + assert.ok(existsSync(result.stateDir)); + const context = JSON.parse(await readFile(f.cliContextPath, "utf8")); + assert.equal(context.configPathsByHost.pi.tokenPilotConfigPath, f.configPath); + }); + it("I3 an existing TokenPilot loader is overwritten without a backup", async () => { + const f = await fixture(); + await f.install(); + const second = await f.install(); + assert.equal(second.backupPath, undefined); + assert.deepEqual((await readdir(join(f.agentDir, "extensions", "tokenpilot"))), ["index.js"]); + }); + it("I4 a foreign loader is backed up and recorded", async () => { + const f = await fixture(); + await mkdir(join(f.agentDir, "extensions", "tokenpilot"), { recursive: true }); + await writeFile(f.loaderPath, "export default () => 'mine';\n"); + const result = await f.install({ now: () => new Date(1_700_000_000_000) }); + assert.equal(result.backupPath, `${f.loaderPath}.bak-1700000000000`); + assert.equal(await readFile(result.backupPath!, "utf8"), "export default () => 'mine';\n"); + const manifest = JSON.parse(await readFile(join(f.agentDir, "tokenpilot-install.json"), "utf8")); + assert.equal(manifest.backupPath, result.backupPath); + }); + it("I5 an existing config is kept byte-identical", async () => { + const f = await fixture(); + await mkdir(f.agentDir, { recursive: true }); + const custom = `${JSON.stringify({ reduction: { maxToolChars: 1800 }, taskStateEstimator: { apiKey: "secret" } })}\n`; + await writeFile(f.configPath, custom); + const result = await f.install(); + assert.equal(result.configCreated, false); + assert.equal(await readFile(f.configPath, "utf8"), custom); + }); + it("I6 re-install keeps the first backup path and configCreated flag", async () => { + const f = await fixture(); + await mkdir(join(f.agentDir, "extensions", "tokenpilot"), { recursive: true }); + await writeFile(f.loaderPath, "foreign\n"); + const first = await f.install(); + await f.install(); + const manifest = JSON.parse(await readFile(join(f.agentDir, "tokenpilot-install.json"), "utf8")); + assert.equal(manifest.backupPath, first.backupPath); + assert.equal(manifest.configCreated, true); + }); + it("I7 the created config contains no credentials", async () => { + const f = await fixture(); + await f.install(); + const raw = JSON.parse(await readFile(f.configPath, "utf8")); + assert.equal(raw.taskStateEstimator.apiKey, undefined); + assert.equal(raw.taskStateEstimator.baseUrl, undefined); + }); +}); + +describe("loader helpers", () => { + it("L1 the loader exports the factory function itself", async () => { + const f = await fixture(); + await f.install(); + const loaded = require(f.loaderPath) as (() => string) & { default: () => string }; + assert.equal(typeof loaded, "function"); + assert.equal(loaded(), "ok"); + assert.equal(loaded.default, loaded); + }); + it("L1b the real built bundle yields the registering factory through the loader", async () => { + const f = await fixture(); + const built = join(__dirname, "..", "dist", "extension.js"); + if (!existsSync(built)) return; // covered by CI after \`pnpm build\` + await f.install({ bundlePath: built }); + const loaded = require(f.loaderPath) as (pi: unknown) => void; + const events: string[] = []; + const tools: string[] = []; + loaded({ on: (event: string) => { events.push(event); }, registerTool: (tool: { name: string }) => { tools.push(tool.name); } }); + assert.deepEqual(events.sort(), ["before_agent_start", "context", "session_shutdown", "session_start", "turn_end"]); + assert.deepEqual(tools, ["memory_fault_recover"]); + }); + it("L2 marker detection", async () => { + const f = await fixture(); + assert.equal(await isTokenPilotPiLoader(f.loaderPath), false); + await f.install(); + assert.equal(await isTokenPilotPiLoader(f.loaderPath), true); + await writeFile(f.loaderPath, "not ours"); + assert.equal(await isTokenPilotPiLoader(f.loaderPath), false); + assert.ok(renderPiLoader("/x").startsWith(PI_LOADER_MARKER)); + }); + it("L3 loaderTarget parses the required path", async () => { + const f = await fixture(); + await f.install(); + assert.equal(loaderTarget(f.loaderPath), f.bundlePath); + await writeFile(f.loaderPath, "garbage"); + assert.equal(loaderTarget(f.loaderPath), undefined); + assert.equal(loaderTarget(join(f.root, "missing.js")), undefined); + }); +}); + +describe("uninstallPiTokenPilot", () => { + it("U1 removes our loader and keeps config and state", async () => { + const f = await fixture(); + const installed = await f.install(); + const result = await uninstallPiTokenPilot({ agentDir: f.agentDir }); + assert.equal(result.loaderRemoved, true); + assert.equal(existsSync(f.loaderPath), false); + assert.ok(existsSync(f.configPath)); + assert.ok(existsSync(installed.stateDir)); + }); + it("U2 leaves a foreign loader untouched", async () => { + const f = await fixture(); + await f.install(); + await writeFile(f.loaderPath, "someone else's"); + const result = await uninstallPiTokenPilot({ agentDir: f.agentDir }); + assert.deepEqual([result.loaderRemoved, result.foreignLoaderKept], [false, true]); + assert.equal(await readFile(f.loaderPath, "utf8"), "someone else's"); + }); + it("U3 restores the recorded backup", async () => { + const f = await fixture(); + await mkdir(join(f.agentDir, "extensions", "tokenpilot"), { recursive: true }); + await writeFile(f.loaderPath, "original\n"); + await f.install(); + const result = await uninstallPiTokenPilot({ agentDir: f.agentDir }); + assert.equal(result.backupRestored, true); + assert.equal(await readFile(f.loaderPath, "utf8"), "original\n"); + }); + it("U4 --purge removes a created config and the state dir", async () => { + const f = await fixture(); + const installed = await f.install(); + const result = await uninstallPiTokenPilot({ agentDir: f.agentDir, purge: true }); + assert.deepEqual([result.configRemoved, result.stateRemoved], [true, true]); + assert.equal(existsSync(f.configPath), false); + assert.equal(existsSync(installed.stateDir), false); + }); + it("U5 --purge keeps a config that predates the install", async () => { + const f = await fixture(); + await mkdir(f.agentDir, { recursive: true }); + await writeFile(f.configPath, "{}\n"); + await f.install(); + const result = await uninstallPiTokenPilot({ agentDir: f.agentDir, purge: true }); + assert.equal(result.configRemoved, false); + assert.ok(existsSync(f.configPath)); + }); + it("U6 uninstall without an install is a no-op", async () => { + const f = await fixture(); + assert.deepEqual(await uninstallPiTokenPilot({ agentDir: f.agentDir }), { + loaderRemoved: false, backupRestored: false, foreignLoaderKept: false, configRemoved: false, stateRemoved: false, + }); + }); + it("U7 install then purge leaves no TokenPilot files behind", async () => { + const f = await fixture(); + await mkdir(join(f.agentDir, "extensions"), { recursive: true }); + await f.install(); + await uninstallPiTokenPilot({ agentDir: f.agentDir, purge: true }); + const remaining = await readdir(f.agentDir, { recursive: true }); + assert.deepEqual(remaining.filter((entry) => /tokenpilot/i.test(String(entry))), []); + }); +}); + +describe("doctor", () => { + async function installed() { + const f = await fixture(); + await f.install(); + const config = await loadTokenPilotPiConfig(f.configPath); + const inspect = (cfg = config) => inspectPiDoctor({ config: cfg, configPath: f.configPath, agentDir: f.agentDir, detectVersion: false }); + return { ...f, config, inspect }; + } + it("D1 a fresh install is healthy", async () => { + const f = await installed(); + const report = await f.inspect(); + assert.deepEqual(report.problems, []); + assert.equal(report.healthy, true); + assert.deepEqual(report.declaredFeatures, ["stabilizer", "reduction", "eviction"]); + }); + it("D2 nothing installed is unhealthy", async () => { + const f = await fixture(); + const configPath = join(f.agentDir, "tokenpilot.json"); + const report = await inspectPiDoctor({ config: normalizeTokenPilotPiConfig({}, { configPath }), configPath, agentDir: f.agentDir, detectVersion: false }); + assert.equal(report.healthy, false); + assert.ok(report.problems.some((p) => p.startsWith("config missing"))); + assert.ok(report.problems.some((p) => p.startsWith("pi loader missing"))); + }); + it("D3 a loader pointing to a missing bundle is reported", async () => { + const f = await installed(); + await rm(f.bundlePath); + const report = await f.inspect(); + assert.ok(report.problems.some((p) => p.startsWith("loader points to a missing bundle"))); + }); + it("D4 a disabled config is reported", async () => { + const f = await installed(); + const report = await f.inspect({ ...f.config, enabled: false }); + assert.ok(report.problems.includes("adapter disabled in config (enabled=false)")); + }); + it("D5 incomplete eviction settings are reported with the missing keys", async () => { + const f = await installed(); + const config = normalizeTokenPilotPiConfig({ stateDir: f.config.stateDir, modules: { eviction: true }, eviction: { enabled: true }, taskStateEstimator: { baseUrl: "http://x" } }); + const report = await f.inspect(config); + assert.ok(report.problems.includes("eviction requested but inactive: estimator_incomplete (missing model, apiKey)")); + }); + it("D6 latest session and activity are reported", async () => { + const f = await installed(); + await writeLatestSessionRef(f.config.stateDir, "sess-9", "2026-09-28T00:00:00.000Z"); + await recordUxEffect(f.config.stateDir, { at: "2026-09-28T00:00:01.000Z", sessionId: "sess-9", model: "qwen3", countMode: "chars", beforeCount: 10, afterCount: 5, savedCount: 5 }); + const report = await f.inspect(); + assert.equal(report.latestSessionId, "sess-9"); + assert.equal(report.latestActivityAt, "2026-09-28T00:00:01.000Z"); + }); + it("D7 the formatted report lists every check", async () => { + const f = await installed(); + const text = formatPiDoctorReport(await f.inspect()); + for (const label of ["tokenpilot config", "enabled", "pi version", "extension loader", "extension bundle", "stateDir", "declared features", "stabilizer", "reduction", "recovery", "eviction", "latest session", "latest activity", "adapter log", "healthy: yes"]) { + assert.ok(text.includes(`- ${label}`), label); + } + }); +}); diff --git a/components/adapters/pi/tests/runtime.test.ts b/components/adapters/pi/tests/runtime.test.ts new file mode 100644 index 00000000..c03d2060 --- /dev/null +++ b/components/adapters/pi/tests/runtime.test.ts @@ -0,0 +1,505 @@ +/** + * Regression matrix: pi/src/extension.ts + pi/src/runtime.ts + * + * Registration + * X1 registers exactly session_start, session_shutdown, before_agent_start, context, + * turn_end and one tool named memory_fault_recover + * X2 the factory does no I/O (config is not loaded until session_start) + * session_start + * S1 loads config, binds the pi session id, writes the latest-session ref + * S2 disabled config → nothing written; later hooks are no-ops + * S3 config loader throws → fails open; later hooks are no-ops + * S4 per-session caches (disclosed-read memo, pending user prefix) are reset + * S5 missing session id → stable fallback id + * before_agent_start + * B1 before session_start / disabled → undefined, options untouched + * B2 no volatile lines → only the recovery section is added + * B3 developer target: volatile lines leave appendSystemPrompt / sections / contextFiles + * (text equals the shared rewrite's forwardedText) and land in tokenpilot_dynamic + * B4 user target: no dynamic section; lines queued for the first user message + * B5 stabilizer off: volatile lines stay; recovery section still added + * B6 TOKENPILOT_DISABLE_RECOVERY_PROTOCOL → no recovery section + * B7 never returns systemPrompt (no forced full-prompt replacement) + * B8 duplicate volatile lines are collected once + * B9 malformed options → fail open, no throw + * context + * C1 disabled → undefined + * C2 nothing to reduce → undefined (pi keeps its own array) + * C3 large tool output → only that toolResult is replaced; text equals the shared + * reduction run directly; a turn binding records total and per-pass savings + * C4 a second identical request returns byte-identical messages + * C5 user target → first real user message prefixed request-locally; input untouched + * C6 reduction off but prefix pending → only the prefix is applied + * C7 malformed messages → undefined, no throw + * C8 a trimmed file read stays byte-identical on the next request in the same + * process after another tool turn is appended (regression: live smoke run sent it + * untrimmed as a "repeat read" → prefix-cache miss) + * C9 a restarted pi process (new runtime, same stateDir + session) sends the + * byte-identical history (archive paths are content-derived) + * C10 disclosed-read memo save failure → the reduced history is still returned, + * warning logged + * turn_end (eviction) + * T1 default config (eviction off) → undefined, estimator never created + * T2 enabled + completed earlier turn → context_edit drafts for evicted entry ids, + * replacement shaped for the target role; current turn untouched + * T3 estimator unavailable → undefined + * T4 empty projection → undefined + * T5 estimator throws → undefined (fail open), warning logged + * T6 host binding without eviction → undefined even when fully configured + * memory_fault_recover tool + * R1 name, description and schema match the shared MCP tool + * R2 known dataKey → archived text and details + * R3 unknown dataKey → not-found text, details.error, no throw + * R4 missing dataKey → "Missing required parameter" + */ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import type { SemanticTaskUpdate, TaskStateEstimator } from "@lightrsi/eviction"; +import { createEmptySessionTaskRegistry, type SessionTaskRegistry } from "@lightrsi/history"; +import { latestSessionPath, loadRecentTurnBindings } from "@lightrsi/host-adapter"; +import { handleMcpRequest } from "@lightrsi/mcp"; +import { rewriteTextForStablePrefix } from "@lightrsi/stabilizer"; + +import { createCanonicalEnvelope } from "../../shared/canonical/before-call.js"; +import { reduceCanonicalEnvelope } from "../../shared/canonical/reduction.js"; +import { adapterLogPath } from "../../shared/canonical/logger.js"; +import { decodePiMessages } from "../src/codec.js"; +import { normalizeTokenPilotPiConfig } from "../src/config.js"; +import { registerTokenPilotPiExtension } from "../src/extension.js"; +import type { + PiAgentMessage, + PiEventMap, + PiExtensionAPI, + PiExtensionContext, + PiProjectedSessionEntry, + PiToolDefinition, +} from "../src/pi-types.js"; +import { PI_DYNAMIC_SECTION, PI_RECOVERY_SECTION, type PiRuntimeDependencies } from "../src/runtime.js"; + +const SESSION = "pi-session-1"; +const BIG = Array.from({ length: 3000 }, (_, i) => `line ${i} output from build step ok`).join("\n"); +const VOLATILE = "Project rules.\nCurrent date: 2026-09-28\nKeep diffs small."; + +type Handler = (event: unknown, ctx: PiExtensionContext) => unknown; + +function fakePi() { + const handlers = new Map(); + const tools: PiToolDefinition[] = []; + const api: PiExtensionAPI = { + on(event, handler) { + handlers.set(event, handler as Handler); + return () => undefined; + }, + registerTool(tool) { + tools.push(tool); + }, + }; + return { api, handlers, tools }; +} + +function ctx(sessionId: string | undefined = SESSION): PiExtensionContext { + return { + cwd: "/work", + sessionManager: { + getCwd: () => "/work", + getSessionId: () => sessionId as string, + getSessionFile: () => undefined, + getBranch: () => [], + }, + model: { id: "qwen3" }, + }; +} + +async function stateDir(): Promise { + return mkdtemp(join(tmpdir(), "tp-pi-")); +} + +async function setup(raw: Record = {}, deps: PiRuntimeDependencies = {}) { + const dir = await stateDir(); + const config = normalizeTokenPilotPiConfig({ stateDir: dir, ...raw }); + const pi = fakePi(); + let loads = 0; + const runtime = registerTokenPilotPiExtension(pi.api, { + loadConfig: async () => { loads += 1; return config; }, + ...deps, + }); + const call = (name: K, event: PiEventMap[K], c = ctx()) => + Promise.resolve(pi.handlers.get(name)!(event, c)) as Promise; + return { dir, config, pi, runtime, call, loads: () => loads }; +} + +function promptEvent(patch: Partial = {}): PiEventMap["before_agent_start"] { + return { + type: "before_agent_start", + prompt: "go", + systemPrompt: "rendered", + systemPromptOptions: { appendSystemPrompt: "", sections: {}, contextFiles: [], ...patch }, + }; +} + +function transcript(output = BIG): PiAgentMessage[] { + return [ + { role: "user", content: "fix the build", timestamp: 1 }, + { role: "assistant", content: [{ type: "toolCall", id: "call_1", name: "bash", arguments: { command: "make" } }], timestamp: 2 }, + { role: "toolResult", toolCallId: "call_1", toolName: "bash", content: [{ type: "text", text: output }], isError: false, timestamp: 3 }, + ]; +} + +const stripArchive = (text: string) => text.replace(/\nArchive: [^\n]*$/, "\nArchive: "); + +describe("registration", () => { + it("X1 registers the five lifecycle handlers and the recovery tool", async () => { + const { pi } = await setup(); + assert.deepEqual([...pi.handlers.keys()].sort(), ["before_agent_start", "context", "session_shutdown", "session_start", "turn_end"]); + assert.deepEqual(pi.tools.map((tool) => tool.name), ["memory_fault_recover"]); + }); + it("X2 the factory does not load config", async () => { + const { loads } = await setup(); + assert.equal(loads(), 0); + }); +}); + +describe("session_start", () => { + it("S1 loads config, binds the session and writes the latest-session ref", async () => { + const env = await setup(); + await env.call("session_start", { type: "session_start", reason: "startup" }); + assert.equal(env.loads(), 1); + assert.equal(env.runtime.sessionId, SESSION); + assert.equal(JSON.parse(await readFile(latestSessionPath(env.dir), "utf8")).sessionId, SESSION); + }); + it("S2 a disabled config writes nothing and later hooks are no-ops", async () => { + const env = await setup({ enabled: false }); + await env.call("session_start", { type: "session_start", reason: "startup" }); + assert.equal(existsSync(latestSessionPath(env.dir)), false); + const event = promptEvent(); + assert.equal(await env.call("before_agent_start", event), undefined); + assert.deepEqual(event.systemPromptOptions.sections, {}); + assert.equal(await env.call("context", { type: "context", messages: transcript() }), undefined); + }); + it("S3 a throwing config loader fails open", async () => { + const env = await setup({}, { loadConfig: async () => { throw new Error("bad json"); } }); + await assert.doesNotReject(env.call("session_start", { type: "session_start", reason: "startup" })); + assert.equal(await env.call("context", { type: "context", messages: transcript() }), undefined); + }); + it("S4 per-session caches are reset", async () => { + const env = await setup(); + const seg = { id: "seg", kind: "volatile", text: "x", priority: 0, metadata: { path: "/a.ts" } } as never; + env.runtime.memo.recordDisclosedReadPaths(["/a.ts"], [seg], new Set(["seg"])); + assert.deepEqual(env.runtime.memo.disclosedReadPaths, ["/a.ts"], "precondition"); + env.runtime.pendingUserPrefix = "stale"; + await env.call("session_start", { type: "session_start", reason: "resume" }); + assert.equal(env.runtime.memo.disclosedReadPaths, undefined); + assert.equal(env.runtime.pendingUserPrefix, undefined); + }); + it("S5 a missing session id falls back to a stable id", async () => { + const env = await setup(); + await env.call("session_start", { type: "session_start", reason: "startup" }, ctx("")); + assert.equal(env.runtime.sessionId, "pi-unknown-session"); + }); +}); + +describe("before_agent_start", () => { + async function started(raw: Record = {}) { + const env = await setup(raw); + await env.call("session_start", { type: "session_start", reason: "startup" }); + return env; + } + it("B1 before session_start the hook is a no-op", async () => { + const env = await setup(); + const event = promptEvent({ appendSystemPrompt: VOLATILE }); + assert.equal(await env.call("before_agent_start", event), undefined); + assert.equal(event.systemPromptOptions.appendSystemPrompt, VOLATILE); + assert.deepEqual(event.systemPromptOptions.sections, {}); + }); + it("B2 without volatile lines only the recovery section is added", async () => { + const env = await started(); + const event = promptEvent({ appendSystemPrompt: "Stable rules." }); + await env.call("before_agent_start", event); + assert.deepEqual(Object.keys(event.systemPromptOptions.sections), [PI_RECOVERY_SECTION]); + assert.ok(event.systemPromptOptions.sections[PI_RECOVERY_SECTION]!.startsWith("[Recovery Protocol]")); + assert.equal(event.systemPromptOptions.appendSystemPrompt, "Stable rules."); + }); + it("B3 developer target moves volatile lines from every input into the dynamic section", async () => { + const env = await started(); + const event = promptEvent({ + appendSystemPrompt: VOLATILE, + sections: { team: "Team notes.\nRequest ID: abc-123" }, + contextFiles: [{ path: "AGENTS.md", content: "Agent rules.\nCurrent date: 2026-09-28" }], + }); + await env.call("before_agent_start", event); + const options = event.systemPromptOptions; + assert.equal(options.appendSystemPrompt, rewriteTextForStablePrefix(VOLATILE).forwardedText); + assert.equal(options.sections.team, rewriteTextForStablePrefix("Team notes.\nRequest ID: abc-123").forwardedText); + assert.equal(options.contextFiles[0]?.content, rewriteTextForStablePrefix("Agent rules.\nCurrent date: 2026-09-28").forwardedText); + assert.equal(options.sections[PI_DYNAMIC_SECTION], "Current date: 2026-09-28\nRequest ID: abc-123"); + assert.equal(env.runtime.pendingUserPrefix, undefined); + }); + it("B4 user target queues the lines for the first user message instead", async () => { + const env = await started({ hooks: { dynamicContextTarget: "user" } }); + const event = promptEvent({ appendSystemPrompt: VOLATILE }); + await env.call("before_agent_start", event); + assert.equal(event.systemPromptOptions.sections[PI_DYNAMIC_SECTION], undefined); + assert.equal(env.runtime.pendingUserPrefix, "Current date: 2026-09-28"); + }); + it("B5 stabilizer off leaves volatile lines but still adds recovery", async () => { + const env = await started({ modules: { stabilizer: false } }); + const event = promptEvent({ appendSystemPrompt: VOLATILE }); + await env.call("before_agent_start", event); + assert.equal(event.systemPromptOptions.appendSystemPrompt, VOLATILE); + assert.deepEqual(Object.keys(event.systemPromptOptions.sections), [PI_RECOVERY_SECTION]); + }); + it("B6 the disable env var removes the recovery section", async () => { + const env = await started(); + process.env.TOKENPILOT_DISABLE_RECOVERY_PROTOCOL = "true"; + try { + const event = promptEvent(); + await env.call("before_agent_start", event); + assert.deepEqual(event.systemPromptOptions.sections, {}); + } finally { + delete process.env.TOKENPILOT_DISABLE_RECOVERY_PROTOCOL; + } + }); + it("B7 the hook never forces a full system prompt", async () => { + const env = await started(); + assert.equal(await env.call("before_agent_start", promptEvent({ appendSystemPrompt: VOLATILE })), undefined); + }); + it("B8 duplicate volatile lines are collected once", async () => { + const env = await started(); + const event = promptEvent({ appendSystemPrompt: VOLATILE, sections: { copy: VOLATILE } }); + await env.call("before_agent_start", event); + assert.equal(event.systemPromptOptions.sections[PI_DYNAMIC_SECTION], "Current date: 2026-09-28"); + }); + it("B9 malformed options fail open", async () => { + const env = await started(); + const event = { type: "before_agent_start", prompt: "x", systemPrompt: "", systemPromptOptions: null } as unknown as PiEventMap["before_agent_start"]; + assert.equal(await env.call("before_agent_start", event), undefined); + assert.match(await readFile(adapterLogPath(env.dir), "utf8"), /before_agent_start failed open/); + }); +}); + +describe("context", () => { + async function started(raw: Record = {}) { + const env = await setup(raw); + await env.call("session_start", { type: "session_start", reason: "startup" }); + return env; + } + it("C1 disabled returns undefined", async () => { + const env = await started({ enabled: false }); + assert.equal(await env.call("context", { type: "context", messages: transcript() }), undefined); + }); + it("C2 nothing to reduce returns undefined", async () => { + const env = await started(); + assert.equal(await env.call("context", { type: "context", messages: transcript("small output") }), undefined); + }); + it("C3 a large tool output is reduced exactly as the shared pipeline would", async () => { + const env = await started(); + const original = transcript(); + const out = await env.call("context", { type: "context", messages: original }) as { messages: PiAgentMessage[] }; + assert.equal(out.messages[0], original[0]); + assert.equal(out.messages[1], original[1]); + assert.notEqual(out.messages[2], original[2]); + const text = ((out.messages[2] as { content: Array<{ text: string }> }).content[0]!).text; + const direct = await reduceCanonicalEnvelope({ + envelope: createCanonicalEnvelope({ hostId: "pi", displayName: "pi", sessionId: SESSION, model: "qwen3", messages: decodePiMessages(original) }), + config: env.config, + }); + const directText = ((direct.envelope.messages[2]!.content as Array<{ text: string }>)[0]!).text; + assert.equal(stripArchive(text), stripArchive(directText)); + const bindings = await loadRecentTurnBindings<{ reductionSavedChars: number; reductionPassSavedChars: Record }>(env.dir, SESSION); + assert.equal(bindings.at(-1)?.reductionSavedChars, BIG.length - text.length); + assert.ok((bindings.at(-1)?.reductionPassSavedChars.tool_payload_trim ?? 0) > 0); + }); + it("C4 a repeated request is byte-identical", async () => { + const env = await started(); + const first = await env.call("context", { type: "context", messages: transcript() }); + await new Promise((resolve) => setTimeout(resolve, 5)); + const second = await env.call("context", { type: "context", messages: transcript() }); + assert.equal(JSON.stringify(second), JSON.stringify(first)); + }); + it("C5 user target prefixes the first real user message request-locally", async () => { + const env = await started({ hooks: { dynamicContextTarget: "user" } }); + await env.call("before_agent_start", promptEvent({ appendSystemPrompt: VOLATILE })); + const original: PiAgentMessage[] = [ + { role: "custom", customType: "note", content: "n", display: true, timestamp: 0 }, + ...transcript("small"), + ]; + const snapshot = JSON.stringify(original); + const out = await env.call("context", { type: "context", messages: original }) as { messages: PiAgentMessage[] }; + assert.equal(out.messages[0], original[0]); + assert.equal((out.messages[1] as { content: string }).content, "Current date: 2026-09-28\n\nfix the build"); + assert.equal(JSON.stringify(original), snapshot); + }); + it("C6 reduction off with a pending prefix only applies the prefix", async () => { + const env = await started({ hooks: { dynamicContextTarget: "user" }, modules: { reduction: false } }); + await env.call("before_agent_start", promptEvent({ appendSystemPrompt: VOLATILE })); + const original = transcript(); + const out = await env.call("context", { type: "context", messages: original }) as { messages: PiAgentMessage[] }; + assert.equal(out.messages[2], original[2]); + assert.ok(String((out.messages[0] as { content: string }).content).startsWith("Current date")); + }); + it("C7 malformed messages never throw", async () => { + const env = await started(); + assert.equal(await env.call("context", { type: "context", messages: null as unknown as PiAgentMessage[] }), undefined); + assert.equal(await env.call("context", { type: "context", messages: [null, 3, "x"] as unknown as PiAgentMessage[] }), undefined); + }); + it("C8 a trimmed file read stays byte-identical on the next request", async () => { + const env = await started(); + const code = Array.from({ length: 400 }, (_, i) => `export function handler${i}(input: string): string {\n return input.trim() + "${i}";\n}\n`).join("\n"); + const base: PiAgentMessage[] = [ + { role: "user", content: "read the handlers", timestamp: 1 }, + { role: "assistant", content: [{ type: "toolCall", id: "call_r", name: "read", arguments: { path: "src/handlers.ts" } }], timestamp: 2 }, + { role: "toolResult", toolCallId: "call_r", toolName: "read", content: [{ type: "text", text: code }], isError: false, timestamp: 3 }, + ]; + const readText = (out: unknown) => JSON.stringify((out as { messages: PiAgentMessage[] }).messages[2]); + const first = await env.call("context", { type: "context", messages: base }); + assert.ok(readText(first).length < JSON.stringify(base[2]).length, "precondition: the read is trimmed"); + const next: PiAgentMessage[] = [ + ...base, + { role: "assistant", content: [{ type: "toolCall", id: "call_w", name: "write", arguments: { path: "out.txt", content: "x" } }], timestamp: 4 }, + { role: "toolResult", toolCallId: "call_w", toolName: "write", content: [{ type: "text", text: "wrote 1 byte" }], isError: false, timestamp: 5 }, + ]; + const second = await env.call("context", { type: "context", messages: next }); + assert.equal(readText(second), readText(first)); + }); + it("C9 a restarted pi process sends byte-identical history", async () => { + const shared = await stateDir(); + const before = await started({ stateDir: shared }); + const first = await before.call("context", { type: "context", messages: transcript() }); + await new Promise((resolve) => setTimeout(resolve, 5)); + const after = await started({ stateDir: shared }); + const second = await after.call("context", { type: "context", messages: transcript() }); + assert.ok(first, "precondition: the first request is reduced"); + assert.equal(JSON.stringify(second), JSON.stringify(first)); + }); + it("C10 a memo save failure still returns the reduced history", async () => { + const env = await started(); + await mkdir(join(env.config.stateDir, "tokenpilot"), { recursive: true }); + await writeFile(join(env.config.stateDir, "tokenpilot", "reduction-memo"), "not a directory"); + // A trimmed file read records a disclosure, so the memo has something to save. + const code = Array.from({ length: 400 }, (_, i) => `export function handler${i}(input: string): string {\n return input.trim() + "${i}";\n}\n`).join("\n"); + const reads: PiAgentMessage[] = [ + { role: "user", content: "read the handlers", timestamp: 1 }, + { role: "assistant", content: [{ type: "toolCall", id: "call_r", name: "read", arguments: { path: "src/handlers.ts" } }], timestamp: 2 }, + { role: "toolResult", toolCallId: "call_r", toolName: "read", content: [{ type: "text", text: code }], isError: false, timestamp: 3 }, + ]; + const out = await env.call("context", { type: "context", messages: reads }) as { messages: PiAgentMessage[] } | undefined; + assert.ok(out && JSON.stringify(out.messages[2]).length < JSON.stringify(reads[2]).length); + assert.match(await readFile(adapterLogPath(env.config.stateDir), "utf8"), /reduction memo save failed/); + }); +}); + +describe("turn_end eviction", () => { + const TASK: SemanticTaskUpdate = { + taskId: "task-build", + objective: "fix the build", + lifecycle: "completed", + coveredTurnAbsIds: [`${SESSION}:t1`], + completionEvidence: ["build passed"], + }; + const EVICTION_ON = { + modules: { eviction: true }, + eviction: { enabled: true, minBlockChars: 256 }, + taskStateEstimator: { baseUrl: "http://127.0.0.1:1/v1", apiKey: "k", model: "m" }, + }; + function memoryStore() { + let current: SessionTaskRegistry = createEmptySessionTaskRegistry(SESSION); + return { load: () => current, persist: (registry: SessionTaskRegistry) => { current = registry; } }; + } + function projection(): PiProjectedSessionEntry[] { + const entry = (id: string, message: PiAgentMessage): PiProjectedSessionEntry => ({ + sourceEntry: { type: "message", id, parentId: null, timestamp: "t" }, + messages: [message], + }); + return [ + entry("e1", { role: "user", content: "fix the build", timestamp: 1 }), + entry("e2", { role: "assistant", content: [{ type: "toolCall", id: "call_1", name: "bash", arguments: {} }], timestamp: 2 }), + entry("e3", { role: "toolResult", toolCallId: "call_1", toolName: "bash", content: [{ type: "text", text: BIG }], isError: false, timestamp: 3 }), + entry("e4", { role: "user", content: "now the docs", timestamp: 4 }), + entry("e5", { role: "toolResult", toolCallId: "call_2", toolName: "bash", content: [{ type: "text", text: BIG }], isError: false, timestamp: 5 }), + ]; + } + const turnEnd = (entries = projection()): PiEventMap["turn_end"] => ({ + type: "turn_end", turnIndex: 1, entries: [], continue: false, context: { contextEntries: entries }, + }); + const estimator = (): TaskStateEstimator => ({ estimate: () => ({ baseVersion: 0, taskUpdates: [TASK] }) }); + + it("T1 default config never creates an estimator", async () => { + let created = false; + const env = await setup({}, { createEstimator: () => { created = true; return estimator(); } }); + await env.call("session_start", { type: "session_start", reason: "startup" }); + assert.equal(await env.call("turn_end", turnEnd()), undefined); + assert.equal(created, false); + }); + it("T2 a completed earlier turn yields context_edit drafts for the evicted entries", async () => { + const env = await setup(EVICTION_ON, { createEstimator: estimator, registryStore: memoryStore() }); + await env.call("session_start", { type: "session_start", reason: "startup" }); + const out = await env.call("turn_end", turnEnd()) as { entries: Array<{ type: string; targetId: string; replacement: { content: unknown } }> }; + assert.deepEqual(out.entries.map((entry) => [entry.type, entry.targetId]), [["context_edit", "e3"]]); + const content = out.entries[0]!.replacement.content as Array<{ type: string; text: string }>; + assert.equal(content[0]?.type, "text"); + assert.ok(content[0]?.text.startsWith("[evicted: tool_result e3]")); + }); + it("T3 an unavailable estimator is a no-op", async () => { + const env = await setup(EVICTION_ON, { createEstimator: () => undefined }); + await env.call("session_start", { type: "session_start", reason: "startup" }); + assert.equal(await env.call("turn_end", turnEnd()), undefined); + }); + it("T4 an empty projection is a no-op", async () => { + const env = await setup(EVICTION_ON, { createEstimator: estimator, registryStore: memoryStore() }); + await env.call("session_start", { type: "session_start", reason: "startup" }); + assert.equal(await env.call("turn_end", turnEnd([])), undefined); + }); + it("T5 a throwing estimator fails open and is logged", async () => { + const env = await setup(EVICTION_ON, { + createEstimator: () => ({ estimate: () => { throw new Error("estimator down"); } }), + registryStore: memoryStore(), + }); + await env.call("session_start", { type: "session_start", reason: "startup" }); + assert.equal(await env.call("turn_end", turnEnd()), undefined); + assert.match(await readFile(adapterLogPath(env.dir), "utf8"), /turn_end failed open Error: estimator down/); + }); + it("T6 a host binding without eviction never evicts", async () => { + const env = await setup(EVICTION_ON, { createEstimator: estimator, registryStore: memoryStore(), supportsEviction: false }); + await env.call("session_start", { type: "session_start", reason: "startup" }); + assert.equal(await env.call("turn_end", turnEnd()), undefined); + }); +}); + +describe("memory_fault_recover tool", () => { + it("R1 matches the shared MCP tool definition", async () => { + const { pi } = await setup(); + const listed = await handleMcpRequest({ jsonrpc: "2.0", id: 1, method: "tools/list" }); + const mcpTool = (listed?.result?.tools as Array>)[0]!; + assert.equal(pi.tools[0]!.name, mcpTool.name); + assert.equal(pi.tools[0]!.description, mcpTool.description); + assert.deepEqual(pi.tools[0]!.parameters, mcpTool.inputSchema); + }); + it("R2 recovers archived content for a known dataKey", async () => { + const env = await setup(); + await env.call("session_start", { type: "session_start", reason: "startup" }); + const reduced = await env.call("context", { type: "context", messages: transcript() }) as { messages: PiAgentMessage[] }; + const text = ((reduced.messages[2] as { content: Array<{ text: string }> }).content[0]!).text; + const dataKey = /"dataKey":"([^"]+)"/.exec(text)![1]!; + const result = await env.pi.tools[0]!.execute("t1", { dataKey }, undefined, undefined, ctx()); + assert.ok(result.content[0]!.text.includes("line 2999 output from build step ok")); + assert.equal((result.details as Record).dataKey, dataKey); + }); + it("R3 an unknown dataKey reports not found without throwing", async () => { + const env = await setup(); + await env.call("session_start", { type: "session_start", reason: "startup" }); + const result = await env.pi.tools[0]!.execute("t1", { dataKey: "nope" }, undefined, undefined, ctx()); + assert.match(result.content[0]!.text, /No archived content found for dataKey: nope/); + assert.equal((result.details as Record).error, "archive_not_found"); + }); + it("R4 a missing dataKey is reported", async () => { + const env = await setup(); + await env.call("session_start", { type: "session_start", reason: "startup" }); + const result = await env.pi.tools[0]!.execute("t1", {}, undefined, undefined, ctx()); + assert.equal(result.content[0]!.text, "Missing required parameter: dataKey"); + }); +}); diff --git a/components/adapters/pi/tsconfig.json b/components/adapters/pi/tsconfig.json new file mode 100644 index 00000000..19420936 --- /dev/null +++ b/components/adapters/pi/tsconfig.json @@ -0,0 +1,14 @@ +{ + "extends": "../../../tsconfig.base.json", + "compilerOptions": { + "types": [ + "node" + ] + }, + "include": [ + "src/**/*.ts", + "scripts/**/*.ts", + "tests/**/*.ts", + "../shared/canonical/**/*.ts" + ] +} diff --git a/components/adapters/shared/cli-context.ts b/components/adapters/shared/cli-context.ts index 282331a8..e059e59b 100644 --- a/components/adapters/shared/cli-context.ts +++ b/components/adapters/shared/cli-context.ts @@ -1,7 +1,7 @@ import { mkdir, readFile, writeFile } from "node:fs/promises"; import { dirname, join } from "node:path"; -type SharedCliHostId = "codex" | "claude-code" | "opencode"; +type SharedCliHostId = "codex" | "claude-code" | "pi" | "opencode"; type SharedCliHostPathOverrides = { tokenPilotConfigPath?: string; diff --git a/components/products/cli/build.ts b/components/products/cli/build.ts index 698c8691..68610e96 100644 --- a/components/products/cli/build.ts +++ b/components/products/cli/build.ts @@ -5,7 +5,7 @@ async function main() { await build({ entryPoints: ["src/cli.ts"], bundle: true, - // adapters/shared/canonical (pulled in by the OpenCode host) has no node_modules + // adapters/shared/canonical (pulled in by the pi/OpenCode hosts) has no node_modules // of its own; resolve its workspace dependencies through an adapter that declares them. nodePaths: [join(__dirname, "node_modules"), join(__dirname, "..", "..", "adapters", "opencode", "node_modules")], outfile: "dist/cli.js", diff --git a/components/products/cli/src/dispatch.ts b/components/products/cli/src/dispatch.ts index 1b46df95..01c89d0a 100644 --- a/components/products/cli/src/dispatch.ts +++ b/components/products/cli/src/dispatch.ts @@ -42,6 +42,10 @@ function currentEnvPathOverrides(host: CliHostId): CliHostPathOverrides | undefi ? { tokenPilotConfigPath, hostConfigPath, hostAuxConfigPath } : undefined; } + if (host === "pi") { + const tokenPilotConfigPath = process.env.TOKENPILOT_PI_CONFIG?.trim(); + return tokenPilotConfigPath ? { tokenPilotConfigPath } : undefined; + } if (host === "opencode") { const tokenPilotConfigPath = process.env.TOKENPILOT_OPENCODE_CONFIG?.trim(); const configDir = process.env.TOKENPILOT_OPENCODE_CONFIG_DIR?.trim(); diff --git a/components/products/cli/src/hosts/factory.ts b/components/products/cli/src/hosts/factory.ts index a083f95b..ec579c1b 100644 --- a/components/products/cli/src/hosts/factory.ts +++ b/components/products/cli/src/hosts/factory.ts @@ -6,6 +6,7 @@ import { createClaudeCodeCleanCommandBackend, createClaudeCodeCliBridge } from " import { createCodexCleanCommandBackend, createCodexCliBridge } from "./codex.js"; import { createOpenClawCleanCommandBackend, createOpenClawCliBridge } from "./openclaw.js"; import { createOpenCodeCliBridge } from "./opencode.js"; +import { createPiCliBridge } from "./pi.js"; import { CLI_HOSTS, getCliHostRegistration, @@ -35,6 +36,9 @@ const CLI_HOST_REGISTRATIONS: CliHostRegistration[] = CLI_HOSTS.map((host) => ({ pathOverrides: target.pathOverrides, }); } + if (host.hostId === "pi") { + return createPiCliBridge({ host: "pi", sessionId: target.sessionId, pathOverrides: target.pathOverrides }); + } if (host.hostId === "opencode") { return createOpenCodeCliBridge({ host: "opencode", sessionId: target.sessionId, pathOverrides: target.pathOverrides }); } diff --git a/components/products/cli/src/hosts/in-process.ts b/components/products/cli/src/hosts/in-process.ts index 3e2737fd..dcb19f5f 100644 --- a/components/products/cli/src/hosts/in-process.ts +++ b/components/products/cli/src/hosts/in-process.ts @@ -1,5 +1,5 @@ /** - * Shared CLI bridge for in-process canonical adapters (currently OpenCode). + * Shared CLI bridge for in-process canonical adapters (pi, OpenCode). * * Mirrors the Claude Code / Codex bridges: flat `tokenpilot.json`, the restricted * host command handler (status, report, doctor, visual, mode conservative|normal, @@ -35,7 +35,7 @@ export const IN_PROCESS_REDUCTION_PASS_NAMES = [ ] as const; export type InProcessHostSpec = { - hostId: "opencode"; + hostId: "pi" | "opencode"; displayName: string; defaultConfigPath(): string; loadConfig(configPath: string): Promise>; diff --git a/components/products/cli/src/hosts/pi.ts b/components/products/cli/src/hosts/pi.ts new file mode 100644 index 00000000..6dff0a65 --- /dev/null +++ b/components/products/cli/src/hosts/pi.ts @@ -0,0 +1,30 @@ +import { + defaultTokenPilotPiConfigPath, + loadTokenPilotPiConfig, + normalizeTokenPilotPiConfig, + writeTokenPilotPiConfig, + type TokenPilotPiConfig, +} from "../../../../adapters/pi/src/config.js"; +import { formatPiDoctorReport, inspectPiDoctor } from "../../../../adapters/pi/src/doctor.js"; +import { piProductSurfaceConfigAdapter, resolvePiStateDir } from "../../../../adapters/pi/src/host-config-adapter.js"; +import type { CliHostPathOverrides } from "../context-store.js"; +import { createInProcessCliBridge, type InProcessHostSpec } from "./in-process.js"; + +export const PI_CLI_HOST_SPEC: InProcessHostSpec = { + hostId: "pi", + displayName: "pi", + defaultConfigPath: defaultTokenPilotPiConfigPath, + loadConfig: async (configPath) => (await loadTokenPilotPiConfig(configPath)) as unknown as Record, + normalizeConfig: (raw, configPath) => normalizeTokenPilotPiConfig(raw, { configPath }) as unknown as Record, + writeConfig: (config, configPath) => writeTokenPilotPiConfig(config as unknown as TokenPilotPiConfig, configPath), + configAdapter: piProductSurfaceConfigAdapter, + resolveStateDir: resolvePiStateDir, + async doctor({ config, configPath }) { + const report = await inspectPiDoctor({ config: config as unknown as TokenPilotPiConfig, configPath }); + return formatPiDoctorReport(report); + }, +}; + +export function createPiCliBridge(target: { host: "pi"; sessionId?: string; pathOverrides?: CliHostPathOverrides }) { + return createInProcessCliBridge(PI_CLI_HOST_SPEC, target); +} diff --git a/components/products/cli/src/hosts/registry.ts b/components/products/cli/src/hosts/registry.ts index 74ffa83a..12c1f7eb 100644 --- a/components/products/cli/src/hosts/registry.ts +++ b/components/products/cli/src/hosts/registry.ts @@ -4,6 +4,7 @@ import { CLAUDE_CODE_PRODUCT_HOST_REGISTRATION } from "../../../../adapters/clau import { CODEX_PRODUCT_HOST_REGISTRATION } from "../../../../adapters/codex/src/product-registration.js"; import { OPENCLAW_PRODUCT_HOST_REGISTRATION } from "../../../../adapters/openclaw/src/product-registration.js"; import { OPENCODE_PRODUCT_HOST_REGISTRATION } from "../../../../adapters/opencode/src/product-registration.js"; +import { PI_PRODUCT_HOST_REGISTRATION } from "../../../../adapters/pi/src/product-registration.js"; import { readCliHostPathOverrides, type CliHostPathOverrides } from "../context-store.js"; export type CliHostRuntime = { @@ -32,6 +33,7 @@ export const CLI_HOSTS = [ OPENCLAW_PRODUCT_HOST_REGISTRATION, CODEX_PRODUCT_HOST_REGISTRATION, CLAUDE_CODE_PRODUCT_HOST_REGISTRATION, + PI_PRODUCT_HOST_REGISTRATION, OPENCODE_PRODUCT_HOST_REGISTRATION, ] as const; @@ -56,9 +58,11 @@ async function productConfigPath(hostId: CliHostId): Promise ? process.env.TOKENPILOT_CODEX_CONFIG?.trim() : hostId === "claude-code" ? process.env.TOKENPILOT_CLAUDE_CODE_CONFIG?.trim() - : hostId === "opencode" - ? process.env.TOKENPILOT_OPENCODE_CONFIG?.trim() - : undefined; + : hostId === "pi" + ? process.env.TOKENPILOT_PI_CONFIG?.trim() + : hostId === "opencode" + ? process.env.TOKENPILOT_OPENCODE_CONFIG?.trim() + : undefined; return environmentPath || (await readCliHostPathOverrides(hostId))?.tokenPilotConfigPath?.trim(); } diff --git a/components/products/cli/src/usage.ts b/components/products/cli/src/usage.ts index 747a4f7e..8d9d4a82 100644 --- a/components/products/cli/src/usage.ts +++ b/components/products/cli/src/usage.ts @@ -34,6 +34,7 @@ export function formatCliUsage(): string { " lightrsi openclaw doctor", " lightrsi openclaw clean --require-tty --session ", " lightrsi claude-code doctor", + " lightrsi pi doctor", " lightrsi opencode report", " lightrsi codex session clean", " lightrsi openclaw session 123e4567-e89b-12d3-a456-426614174000 report", diff --git a/components/products/cli/tests/in-process-hosts.test.ts b/components/products/cli/tests/in-process-hosts.test.ts index d08b36db..6213eeb8 100644 --- a/components/products/cli/tests/in-process-hosts.test.ts +++ b/components/products/cli/tests/in-process-hosts.test.ts @@ -1,12 +1,12 @@ /** - * Regression matrix: OpenCode registration in the shared CLI - * (hosts/in-process.ts, hosts/opencode.ts, registry, factory, dispatch, usage) + * Regression matrix: pi + OpenCode registration in the shared CLI + * (hosts/in-process.ts, hosts/pi.ts, hosts/opencode.ts, registry, factory, dispatch, usage) * * Registration - * H1 CLI_HOSTS lists opencode after the existing hosts (order preserved) - * H2 parseCliHostId accepts "opencode" - * H3 usage lists the host and an example - * Command surface + * H1 CLI_HOSTS lists pi and opencode after the existing hosts (order preserved) + * H2 parseCliHostId accepts "pi" and "opencode" + * H3 usage lists both hosts and an example for each + * Command surface (each case runs for both hosts) * C1 status prints every shared key for the host's config * C2 mode conservative / normal write the standard presets; aggressive refused; * anything else prints usage @@ -17,7 +17,7 @@ * C6 doctor returns the host's own doctor report * C7 report: no activity → "No TokenPilot session stats yet."; with activity → * report for the latest session - * C8 the TOKENPILOT_OPENCODE_CONFIG env override selects the config file + * C8 env overrides (TOKENPILOT_PI_CONFIG / TOKENPILOT_OPENCODE_CONFIG) select the config file */ import assert from "node:assert/strict"; import { after, describe, it } from "node:test"; @@ -34,6 +34,7 @@ import { formatCliUsage } from "../src/usage.js"; const saved = { HOME: process.env.HOME, USERPROFILE: process.env.USERPROFILE, + TOKENPILOT_PI_CONFIG: process.env.TOKENPILOT_PI_CONFIG, TOKENPILOT_OPENCODE_CONFIG: process.env.TOKENPILOT_OPENCODE_CONFIG, TOKENPILOT_OPENCODE_CONFIG_DIR: process.env.TOKENPILOT_OPENCODE_CONFIG_DIR, }; @@ -45,6 +46,7 @@ after(() => { }); const HOSTS = [ + { id: "pi", env: "TOKENPILOT_PI_CONFIG", doctorTitle: "TokenPilot pi doctor:", statusTitle: "TokenPilot pi status:" }, { id: "opencode", env: "TOKENPILOT_OPENCODE_CONFIG", doctorTitle: "TokenPilot OpenCode doctor:", statusTitle: "TokenPilot OpenCode status:" }, ] as const; @@ -62,14 +64,15 @@ const readConfig = async (path: string) => JSON.parse(await readFile(path, "utf8 describe("registration", () => { it("H1 hosts are appended in order", () => { - assert.deepEqual(CLI_HOSTS.map((h) => h.hostId), ["openclaw", "codex", "claude-code", "opencode"]); + assert.deepEqual(CLI_HOSTS.map((h) => h.hostId), ["openclaw", "codex", "claude-code", "pi", "opencode"]); }); it("H2 host ids parse", () => { + assert.equal(parseCliHostId("pi"), "pi"); assert.equal(parseCliHostId("opencode"), "opencode"); }); - it("H3 usage lists the host", () => { + it("H3 usage lists both", () => { const usage = formatCliUsage(); - for (const text of [" opencode\n", "lightrsi opencode report"]) assert.ok(usage.includes(text), text); + for (const text of [" pi\n", " opencode\n", "lightrsi pi doctor", "lightrsi opencode report"]) assert.ok(usage.includes(text), text); }); }); diff --git a/docs/adapters/pi-design.md b/docs/adapters/pi-design.md new file mode 100644 index 00000000..c86b05b9 --- /dev/null +++ b/docs/adapters/pi-design.md @@ -0,0 +1,168 @@ +# TokenPilot adapter design: pi + +Status: implemented in `components/adapters/pi/`. This note was written before implementation and updated afterwards to match what was built. + +## Pinned versions + +| Item | Value | +| :-- | :-- | +| LightRSI base commit | `3b7f28d76c98de85999cf94ae99d39ad144fa327` | +| pi package | `@earendil-works/pi-coding-agent@0.87.1` (npm `latest` on 2026-09-28) | +| pi runtime deps checked | `@earendil-works/pi-ai@0.87.1`, `@earendil-works/pi-agent-core@0.87.1` | +| Sources checked | the published package's `dist/core/extensions/types.d.ts`, `dist/core/session-manager.d.ts`, `dist/core/system-prompt.{d.ts,js}`, `dist/core/messages.d.ts`, `docs/extensions.md`, `docs/compaction.md`, `docs/session-format.md` | + +Every "verified" statement below cites one of those files. pi ships its type +declarations and docs inside the npm tarball, so the pinned version's contract +can be read without cloning the monorepo. + +## Design First (playbook questions) + +| Question | Answer for pi 0.87.1 | Evidence | +| :-- | :-- | :-- | +| Can the host rewrite requests before model execution? | Yes, in process. `context` returns replacement messages for one LLM call; `context_with_system` owns the full transcript; `before_provider_request` can replace the raw provider payload; `before_agent_start` can mutate structured prompt sections. | `types.d.ts` `ContextEvent`, `ContextWithSystemEvent`, `BeforeProviderRequestEvent`, `BeforeAgentStartEvent` | +| Can it rewrite responses after model execution? | Yes. `message_end` can replace a finalized message (same role). `tool_result` handlers can replace tool results before they enter the session. | `MessageEndEventResult`, `ToolResultEventResult` | +| Streaming chunks, final responses, or both? | Both: `message_update` (token stream, notification only) and `message_end` (final, replaceable). | `MessageUpdateEvent`, `MessageEndEvent` | +| Transcript history exposed directly, or reconstructed? | Directly. `context` receives the model-bound `AgentMessage[]`; `ctx.sessionManager.getBranch()` returns the active branch's session entries. | `ContextEvent`, `ReadonlySessionManager` | +| File-, hook-, plugin- or API-based? | Plugin (in-process TypeScript extension loaded through `jiti`) plus native tool registration. No proxy is needed. | `docs/extensions.md` | + +## Host facts verified against 0.87.1 + +| Brief claim | Result | Evidence | +| :-- | :-- | :-- | +| `before_agent_start` exposes structured `systemPromptOptions`; changing sections lets pi append a delta instead of replacing the prompt | Confirmed. `systemPromptOptions: NormalizedBuildSystemPromptOptions` is mutable; returning `systemPrompt` forces a full replacement. Sections are rendered by `buildSystemPromptSections` and diffed by `diffSystemPromptSections`. | `types.d.ts` `BeforeAgentStartEvent` / `BeforeAgentStartEventResult`; `system-prompt.js` | +| `context` / `context_with_system` are request-local transforms | Confirmed. `context` excludes system messages and pi restores prompt/tool state afterwards. | `types.d.ts` doc comments on `ContextEvent` / `ContextWithSystemEvent` | +| `tool_result` handlers compose and can replace a result before it enters the session | Confirmed (`content`, `details`, `isError`, `usage`). | `ToolResultEventResult`, `docs/extensions.md` "Events and concurrency" | +| `tool_call` can mutate input or block | Confirmed. A `tool_call` handler failure blocks the tool (fail-safe, not fail-open). | `ToolCallEventResult`, `docs/extensions.md` "Errors and cleanup" | +| `turn_end` / `agent_before_settle` can return `context_edit` or `compaction` entries | Confirmed (`SessionBoundaryDraft = custom | custom_message | context_edit | compaction`). `context_edit.replacement` is `{content}` or `null` (omit from context). | `BoundaryResult`, `ContextEditEntryDraft`, `session-manager.d.ts` `ContextEditEntry` | +| `session_before_compact` can cancel or supply a custom summary | Confirmed (`cancel`, `compaction`). Event carries `reason: manual|threshold|overflow`. | `SessionBeforeCompactEvent`, `SessionBeforeCompactResult` | +| `pi.registerTool()`; does pi support MCP natively? | `registerTool` confirmed. **No native MCP**: neither the docs nor the extension types mention MCP. | `docs/*.md` (no MCP references), `types.d.ts` | +| `pi.appendEntry()` and `getBranch()`; sessions branch | Confirmed. Docs say to rebuild branch-sensitive state from `getBranch()` at `session_start`. | `docs/extensions.md` "State" | +| Do not start long-lived resources in the factory | Confirmed. | `docs/extensions.md` "Respect the runtime lifecycle" | +| Handler errors | pi "reports handler errors and continues where possible" — except `tool_call`, which blocks. The adapter still catches everything itself so a TokenPilot failure is never surfaced as a host error. | `docs/extensions.md` "Errors and cleanup" | + +Additional facts that shaped the design: + +- The default pi system prompt has no volatile lines (no date/time). `cwd` is its own + `` section. pi already keeps the provider prefix stable by recording the + structured prompt once and appending section deltas. + (`system-prompt.js` `buildSystemPromptSections`.) +- Session identity: `ctx.sessionManager.getSessionId()`, `getSessionFile()`, `getCwd()`. + Turn identity: `turn_start.turnIndex`. (`session-manager.d.ts`, `TurnStartEvent`.) +- Agent directory: `~/.pi/agent` (override `PI_CODING_AGENT_DIR`); user extensions + load from `/extensions/` as a file or a directory with `index.{ts,js}`. + (`docs/configuration.md`, `docs/extensions.md`.) + +## Surface table + +| Surface | pi 0.87.1 | Closest existing adapter precedent | +| :-- | :-- | :-- | +| Install / uninstall flow | `install:pi` writes a marker-tagged loader `~/.pi/agent/extensions/tokenpilot/index.js` that imports the built `dist/extension.js`, plus runtime config `~/.pi/agent/tokenpilot.json` (normal mode). No pi-owned config file is modified. `uninstall:pi` removes the loader only if it carries the TokenPilot marker; an existing non-TokenPilot file at that path is backed up (`.bak-`) before install and restored on uninstall. | Claude Code / Codex `install.ts` (runtime config file next to host config; reversible) | +| Session id, turn id, workspace root | `ctx.sessionManager.getSessionId()`; `turn_start.turnIndex` (counter per session); `ctx.cwd`. | Claude Code hook `SessionStart` + session snapshot | +| Transcript bridge (host ↔ canonical) | `AgentMessage[]` ↔ `HostRequestEnvelope.messages` (`RuntimeMessage`). `user`→`user` (text/image blocks); `assistant` text→`text`, `toolCall`→`tool_call`, `thinking`→ preserved verbatim in message metadata (the kernel has no reasoning block); `toolResult`→`tool` role with one `tool_result` block; `custom`, `bashExecution`, `branchSummary`, `compactionSummary` → pass-through messages (never rewritten). Encode applies canonical tool-result text back onto the original host object, so an unchanged envelope round-trips byte-for-byte. | Claude Code `messages-codec.ts` (host payload ↔ envelope) | +| Stable-prefix path | `before_agent_start`: run the shared stabilizer (`rewriteTextForStablePrefix`) over each mutable prompt input (`appendSystemPrompt`, custom `sections`, context-file contents; pi hands each run a fresh normalised copy). Target `developer` (default): volatile lines move to a trailing `tokenpilot_dynamic` section so pi emits a section delta instead of a full prompt replacement. Target `user`: they are prepended request-locally to the first user message in the `context` hook, which is the shared stabilizer's user semantics; a persisted custom message would grow the transcript every turn. `systemPrompt`/`forceSystemPrompt` are never used. The shared recovery protocol is added as a stable `tokenpilot_recovery` section. The reference adapters get it from the shared before-call injector, which appends to `instructions`, and pi's message hook cannot carry that. The visual snapshot comes from the shared `buildStabilityVisualSnapshotFromEnvelopes`. | OpenClaw `root-prompt-stabilizer.ts` (shared `rewriteTextForStablePrefix`) | +| Reduction path | `context` (request-local, fires before every LLM call): decode → canonical envelope → shared before-call pipeline (`prepareObservedBeforeCall` → `@lightrsi/reduction` analyzers + `runReductionBeforeCall`) → encode the reduced tool-result text back. Same position in the request lifecycle as the Codex/Claude Code proxies and OpenClaw's request-time reduction. | Claude Code `reduction.ts` + `prepareObservedBeforeCall` | +| Eviction path (opt-in) | `turn_end`: the projected branch (`event.context.contextEntries`) → canonical surface entries → shared `runCanonicalEvictionCycle` (`@lightrsi/eviction` canonical-surface core, lifted from DSH): estimator → registry (durable, CAS) → independent safety policy → returned `context_edit` drafts that replace the evicted entry's content with a stub plus a recovery hint (originals archived). `turn_end` drafts commit before pi's threshold compaction check (`agent-session.js`: `_checkCompaction` runs before `_runBeforeSettleBoundary`, so `agent_before_settle` would be too late). Off by default; needs `modules.eviction`, `eviction.enabled` and `taskStateEstimator.{baseUrl,model,apiKey}`. | DeepSeek Harness `eviction-engine.ts` / `eviction-cycle.ts` (same shared cycle) | +| Recovery path | pi has no MCP, so `pi.registerTool({ name: "memory_fault_recover" })` exposes the same tool name, description and JSON schema as the shared MCP server and calls the shared `resolveMemoryFaultRecover` from `@lightrsi/mcp` (the function behind the MCP server's `tools/call`). | Codex / Claude Code register the shared MCP server; OpenClaw registers a native tool | +| Runtime state root, namespace, archive path | Config `~/.pi/agent/tokenpilot.json` (override `TOKENPILOT_PI_CONFIG`). State `~/.pi/agent/tokenpilot-state/tokenpilot/`, namespace `tokenpilot`, archives `/tokenpilot/tool-result-archives//` via `configureStatePathResolver`. | Claude Code (`~/.claude/tokenpilot-state/tokenpilot/`) | +| Status, report, doctor, visual | Shared CLI registration: `lightrsi pi status|report|doctor|visual|mode|stabilizer|reduction`. Report/visual read `ux-effects` written by the shared observability path. `doctor` checks config, loader file, built bundle, state dir, recovery tool wiring, and last activity. | Claude Code `products/cli/src/hosts/claude-code.ts` | +| Declared feature subset (`host-binding.ts`) | `stabilizer`, `reduction`, `eviction` (eviction opt-in). | Codex, Claude Code, OpenClaw | +| Supported modes | `conservative`, `normal` (default). `aggressive` refused (requires eviction). | Codex, Claude Code | + +## Behavioural decisions + +### Reduction runs at request time, not at `tool_result` + +The brief suggests `tool_result` for reduction. All three reference adapters run +reduction at request time over the whole outgoing history (Codex/Claude Code proxies, +OpenClaw's default path; OpenClaw's persistent `tool_result_persist` is opt-in and off +by default). Request-time reduction is also what the `read_state_compaction` pass needs: +it compacts an earlier read once a later read supersedes it, which a single +`tool_result` event cannot see. Using `tool_result` would therefore change what +reduction does, which the fidelity rule forbids. `tool_result` is not hooked. + +Consequence: pi's session file keeps the full original tool output; only the model +request sees reduced text. Native pi compaction therefore summarises the original +outputs (correct, but it does not benefit from reduction). + +### Byte-stable history across requests + +Reduction re-runs over the whole history on every request, so the same tool output must +produce exactly the same text each time or the provider's prefix cache misses from that +point on. This is most visible on llama-server, whose cache is a plain longest-common-prefix +match. The shared `artifact-store` names archives by content (#90), so a re-reduced +result keeps the same `Archive:` line on every request and across pi restarts (every +`pi -p` call, `--continue`, `/resume`). The adapter adds nothing for this. Regression +test: `runtime` C9. + +### Repeat-read detection across requests + +`tool_payload_trim` leaves a file read untrimmed when its path was already disclosed: +the model re-reading a file it has seen summarised is taken as a request for the full +body. The pass reports the disclosed paths in `disclosedReadPaths` so a caller can carry +them forward. pi resends the whole history on every request, so carrying every path +makes the pass treat the *same* read as its own repeat on the next request and send it +untrimmed. The first live smoke run hit exactly this: the `data.json` read went from +1,100 chars back to 50,707 mid-turn, and the prefix cache missed from there on (the +same bug fixed for Claude Code in #92). + +The shared `ReductionMemo` (also used by OpenCode) therefore records which read disclosed +each path, and carries a path only once that read has left the history (compaction, +eviction). As in #92, only a `read`/`file_read` result actually trimmed in that request +can own a path, and the record is rebuilt from the pass's bounded reported set. A +disclosure whose read is still present is re-detected by the pass from the history +itself, so output is a pure function of the history and genuine repeat reads behave as +in the reference. + +The memo is saved per session to `/tokenpilot/reduction-memo/.json` +when it changes and loaded at `session_start`, so a restarted pi still honours a +deliberate re-read. A missing or unreadable file yields an empty memo. Regression tests: +`runtime` C8 and C10 here; the shared memo itself is covered by the OpenCode package's +`shared-canonical-reduction` and `shared-canonical-memo-store` suites. + +### Segment identity + +Claude Code uses message-index segment ids (`message--block-`). Index ids shift +whenever pi compacts or a branch changes, which would change archive names and memo +owners between requests. The pi +adapter uses `tool-` for tool-result segments (tool-call ids are stable in +pi sessions) and falls back to index ids only when an id is missing. Segment ids are +opaque to the shared passes. They are used only for archive file names and bindings. + +### Fail-open + +Every hook body runs inside a guard. On any exception it returns `undefined` (pi +proceeds unmodified) and appends a line to `/tokenpilot/adapter.log`. Nothing +is written to stdout or stderr, because pi's TUI owns the terminal. Malformed messages +(unknown roles, non-array content) are passed through untouched by the codec. The +recovery tool reports "not found" as an ordinary text result, not a thrown error, which +matches the MCP server's `isError` result rather than a transport failure. + +### Branching + +The adapter keeps no branch-sensitive state in the session. Archive names are derived from +content, and the disclosed-read memo is rebuilt from the history on every request, so both +are valid on every branch. Per-session caches are reset on +`session_start` (reasons `startup|reload|new|resume|fork`) and released on +`session_shutdown`. Nothing is started in the extension factory. + +## Gaps against the reference (OpenClaw) and how each is handled + +| Gap | Handling | +| :-- | :-- | +| Eviction needed shared primitives that lived in the DSH adapter | Solved by #95: DSH's tool closure, safety policy, registry classification, eviction cycle and estimator construction moved into `@lightrsi/eviction/src/canonical-surface/` with generic ids. pi uses the same cycle as OpenCode. `eviction on/off/set minBlockChars` are exposed as on Claude Code. | +| Eviction during overflow recovery | `session_before_compact` with `reason: overflow` fires mid-run, where no boundary can append entries, so eviction does not run there. It runs at every `turn_end` before that point is reached. | +| `mode aggressive` | Not exposed, as on Claude Code: it would have to turn on eviction without the estimator credentials that eviction needs. Enable eviction explicitly. Open question for a human. | +| No MCP in pi | Recovery is a native tool backed by the shared `resolveMemoryFaultRecover`. The tool name and schema are identical to the MCP tool. | +| Reduction is request-local, so pi's persisted session keeps full outputs | Same as the Codex/Claude Code proxies. Documented above. | +| No in-host slash commands | Same as Codex/Claude Code: the shared `lightrsi pi ...` CLI is the control surface. | +| Stabilizer is mostly a no-op on the default pi prompt | pi already emits section deltas, and its default prompt contains no volatile lines. The stabilizer acts only when user or project prompt inputs carry volatile lines (dates, request ids, ...). This is reported in `status` and the visual, not hidden. | +| `prompt_cache_key` routing (Codex/Claude Code) | Not applicable to local OpenAI-compatible servers, which cache by prefix. Not emitted. | +| Context Cleaner | Out of scope. See below. | + +## Context Cleaner (future) + +pi can host the Cleaner natively: `pi.registerCommand("tokenpilot-clean")` for the +analyse/select flow, a durable schedule via `pi.appendEntry("tokenpilot-cleaner", …)` +(branch-aware), and the one allowed canonical rewrite as `context_edit` entries returned +from the next `agent_before_settle`. The DSH adapter's `cleaner-pre-step.ts` is the +closest precedent. It depends on the same shared eviction primitives listed above. diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 2608d242..d327d72b 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -238,6 +238,49 @@ importers: specifier: ^5.9.3 version: 5.9.3 + components/adapters/pi: + dependencies: + '@lightrsi/artifact-store': + specifier: workspace:* + version: link:../../packages/foundation/artifact-store + '@lightrsi/eviction': + specifier: workspace:* + version: link:../../packages/features/eviction + '@lightrsi/history': + specifier: workspace:* + version: link:../../packages/foundation/history + '@lightrsi/host-adapter': + specifier: workspace:* + version: link:../../packages/foundation/host-adapter + '@lightrsi/kernel': + specifier: workspace:* + version: link:../../packages/foundation/kernel + '@lightrsi/mcp': + specifier: workspace:* + version: link:../../products/mcp + '@lightrsi/product-surface': + specifier: workspace:* + version: link:../../packages/foundation/product-surface + '@lightrsi/reduction': + specifier: workspace:* + version: link:../../packages/features/reduction + '@lightrsi/stabilizer': + specifier: workspace:* + version: link:../../packages/features/stabilizer + '@lightrsi/tokenpilot': + specifier: workspace:* + version: link:../../presets/tokenpilot + devDependencies: + esbuild: + specifier: ^0.25.10 + version: 0.25.12 + tsx: + specifier: ^4.21.0 + version: 4.21.0 + typescript: + specifier: ^5.9.3 + version: 5.9.3 + components/packages/features/cleaner: dependencies: '@lightrsi/history': diff --git a/website/.vitepress/config.ts b/website/.vitepress/config.ts index e768adcf..5742ba64 100644 --- a/website/.vitepress/config.ts +++ b/website/.vitepress/config.ts @@ -113,6 +113,7 @@ export default defineConfig({ { text: 'Codex', link: '/hosts/codex' }, { text: 'Claude Code', link: '/hosts/claude-code' }, { text: 'DeepSeek Harness', link: '/hosts/deepseek-harness' }, + { text: 'pi', link: '/hosts/pi' }, { text: 'OpenCode', link: '/hosts/opencode' }, ], }, diff --git a/website/hosts/compatibility.md b/website/hosts/compatibility.md index 781dfcc9..01dbbe44 100644 --- a/website/hosts/compatibility.md +++ b/website/hosts/compatibility.md @@ -1,6 +1,6 @@ # Host Compatibility -TokenPilot integrates with OpenClaw, Codex CLI, Claude Code, DeepSeek Harness, and OpenCode. Each integration has a different boundary, so capabilities and setup instructions are documented per host. +TokenPilot integrates with OpenClaw, Codex CLI, Claude Code, DeepSeek Harness, pi, and OpenCode. Each integration has a different boundary, so capabilities and setup instructions are documented per host. ## Supported Hosts @@ -10,6 +10,7 @@ TokenPilot integrates with OpenClaw, Codex CLI, Claude Code, DeepSeek Harness, a | [Codex CLI](./codex) | Local proxy + hooks | `components/adapters/codex/` | | [Claude Code](./claude-code) | Local gateway + MCP | `components/adapters/claude-code/` | | [DeepSeek Harness](./deepseek-harness) | Native Cordis plugin + optional context eviction + session status | `components/adapters/deepseek-harness/` | +| [pi](./pi) | In-process extension + native recovery tool | `components/adapters/pi/` | | [OpenCode](./opencode) | In-process v1 plugin + recovery MCP | `components/adapters/opencode/` | DeepSeek Harness provides opt-in context eviction and the read-only `/tokenpilot-status` command. It uses Harness's own transport and compaction, with separate profile configuration. See the [DeepSeek Harness guide](./deepseek-harness) for installation, required settings, and troubleshooting. @@ -18,24 +19,24 @@ The matrix below distinguishes shared product interfaces from host-native capabi ## TokenPilot Host Feature Matrix -| Feature | OpenClaw | Codex | Claude Code | DeepSeek Harness | OpenCode | -| :-- | :-- | :-- | :-- | :-- | :-- | -| Stable Prefix | ✅ | ✅ | ✅ | — | ✅ (`developer` target only) | -| Context Reduction | ✅ | ✅ | ✅ | — | ✅ | -| Automatic Context Eviction | ✅ | Opt-in response-chain rebase¹ | — | Opt-in; estimator and durable state required | Opt-in; durable request overlay³ | -| User-approved [Context Cleaner](/user-guide/context-cleaner) | Scheduled canonical apply | Scheduled response-chain rebase | Scheduled request overlay | Native `/tokenpilot-clean`; scheduled surface transaction | — | -| Visual Inspector | ✅ | ✅ | ✅ | — | ✅ | -| Session Reports (`report`) | ✅ | ✅ | ✅ | —; native status instead | ✅ | -| Session Status | `/lightrsi status` or CLI | `lightrsi codex status` | `lightrsi claude-code status` | `/tokenpilot-status`² | `lightrsi opencode status` | -| Shared Standalone CLI | ✅ | ✅ | ✅ | —; Harness plugin interface | ✅ | -| MCP Recovery Server | ✅ | ✅ | ✅ | — | ✅ | -| `mode conservative` | ✅ | ✅ | ✅ | —; profile configuration | ✅ | -| `mode normal` | ✅ | ✅ | ✅ | —; profile configuration | ✅ | -| `mode aggressive` | ✅ | — | — | —; profile configuration | — | -| Proxy / Gateway Startup | Gateway restart | SessionStart hook | SessionStart hook | Not needed; Harness owns transport | Not needed; in-process | +| Feature | OpenClaw | Codex | Claude Code | DeepSeek Harness | pi | OpenCode | +| :-- | :-- | :-- | :-- | :-- | :-- | :-- | +| Stable Prefix | ✅ | ✅ | ✅ | — | ✅ (prompt sections) | ✅ (`developer` target only) | +| Context Reduction | ✅ | ✅ | ✅ | — | ✅ | ✅ | +| Automatic Context Eviction | ✅ | Opt-in response-chain rebase¹ | — | Opt-in; estimator and durable state required | Opt-in; native `context_edit`³ | Opt-in; durable request overlay³ | +| User-approved [Context Cleaner](/user-guide/context-cleaner) | Scheduled canonical apply | Scheduled response-chain rebase | Scheduled request overlay | Native `/tokenpilot-clean`; scheduled surface transaction | — | — | +| Visual Inspector | ✅ | ✅ | ✅ | — | ✅ | ✅ | +| Session Reports (`report`) | ✅ | ✅ | ✅ | —; native status instead | ✅ | ✅ | +| Session Status | `/lightrsi status` or CLI | `lightrsi codex status` | `lightrsi claude-code status` | `/tokenpilot-status`² | `lightrsi pi status` | `lightrsi opencode status` | +| Shared Standalone CLI | ✅ | ✅ | ✅ | —; Harness plugin interface | ✅ | ✅ | +| MCP Recovery Server | ✅ | ✅ | ✅ | — | —; native `memory_fault_recover` tool (pi has no MCP) | ✅ | +| `mode conservative` | ✅ | ✅ | ✅ | —; profile configuration | ✅ | ✅ | +| `mode normal` | ✅ | ✅ | ✅ | —; profile configuration | ✅ | ✅ | +| `mode aggressive` | ✅ | — | — | —; profile configuration | — | — | +| Proxy / Gateway Startup | Gateway restart | SessionStart hook | SessionStart hook | Not needed; Harness owns transport | Not needed; in-process | Not needed; in-process | ¹ The [Codex adapter README](https://github.com/zjunlp/LightRSI/blob/main/components/adapters/codex/README.md#task-state-estimator-bridge-pr-b) documents opt-in automatic eviction with both `taskStateEstimator.enabled` and `contextRewrite.enabled`. This does not provide the OpenClaw `eviction` commands or `aggressive` mode. ² DeepSeek Harness registers its status command when the profile provides command and session projection services. Its eviction pass runs before native Harness compaction by default. See [DeepSeek Harness](/hosts/deepseek-harness) for configuration and status interpretation. -³ OpenCode eviction uses the shared canonical-surface core (`@lightrsi/eviction`, lifted from the DeepSeek Harness adapter). It needs `modules.eviction`, `eviction.enabled` and `taskStateEstimator.{baseUrl,model,apiKey}`. Evicted content is archived and recoverable with `memory_fault_recover`. +³ pi and OpenCode eviction uses the shared canonical-surface core (`@lightrsi/eviction`, lifted from the DeepSeek Harness adapter). It needs `modules.eviction`, `eviction.enabled` and `taskStateEstimator.{baseUrl,model,apiKey}`. Evicted content is archived and recoverable with `memory_fault_recover`. diff --git a/website/hosts/pi.md b/website/hosts/pi.md new file mode 100644 index 00000000..cd0f1b1c --- /dev/null +++ b/website/hosts/pi.md @@ -0,0 +1,52 @@ +# pi + +The pi integration is an **in-process extension**. TokenPilot runs inside [pi](https://pi.dev)'s own request lifecycle, so there is no proxy or gateway, and no pi-owned config file is modified. Verified on pi `0.87.1`. + +## Installation + +```bash +pnpm install +npm --prefix components/adapters/pi run build +npm --prefix components/products/cli run build +npm --prefix components/adapters/pi run install:pi +``` + +This command: + +- writes a marker-tagged loader at `~/.pi/agent/extensions/tokenpilot/index.js`, backing up any foreign file at that path +- writes `~/.pi/agent/tokenpilot.json` in `normal` mode if it does not exist (never credentials) +- installs the `lightrsi` launcher + +`PI_CODING_AGENT_DIR` and `TOKENPILOT_PI_CONFIG` override the locations. Remove with `npm --prefix components/adapters/pi run uninstall:pi` (add `-- --purge` to also delete the created config and state). + +## How it works + +| TokenPilot feature | pi hook | +| :-- | :-- | +| Stable prefix | `before_agent_start`: volatile prompt lines move to a trailing `tokenpilot_dynamic` section, so pi appends a section delta instead of replacing the prompt | +| Reduction | `context`: request-time, over the whole history | +| Recovery | native `memory_fault_recover` tool (pi has no MCP), same schema as the shared MCP tool | +| Eviction (opt-in) | `turn_end` → native `context_edit` entries, committed before pi's threshold compaction | + +## Verification + +1. Start pi (or `/reload`). +2. `lightrsi pi doctor` +3. After a few turns: `lightrsi pi report` or `lightrsi pi visual` + +## Commands + +```bash +lightrsi pi status | report | doctor | visual +lightrsi pi mode conservative|normal +lightrsi pi stabilizer on|off|target developer|user +lightrsi pi reduction on|off|pass on|off +lightrsi pi eviction status|on|off|set minBlockChars +``` + +## Troubleshooting + +- Hook failures never break pi. They are logged to `~/.pi/agent/tokenpilot-state/tokenpilot/tokenpilot/adapter.log`. +- `eviction: inactive (estimator_incomplete)` in `doctor` means `taskStateEstimator.baseUrl`, `model` or `apiKey` is missing. + +See the [adapter README](https://github.com/zjunlp/LightRSI/blob/main/components/adapters/pi/README.md) and the [design note](https://github.com/zjunlp/LightRSI/blob/main/docs/adapters/pi-design.md).