diff --git a/docs/key-rotation.md b/docs/key-rotation.md new file mode 100644 index 000000000000..520331a5b60d --- /dev/null +++ b/docs/key-rotation.md @@ -0,0 +1,116 @@ +# API Key Rotation & Throttle Suppression + +## 用户使用方式 + +### 配置多个 API Key + +在 `OPENCODE_API_KEY` 中用英文逗号分隔多个 key： + +```bash +export OPENCODE_API_KEY="sk-key1,sk-key2,sk-key3" +opencode run "帮我写一个排序函数" +``` + +opencode 会依次从 key1 开始尝试。遇到限额（429）或无效 key（401）时自动切换到下一个，对用户完全透明。 + +### 相关环境变量 + +| 变量 | 默认值 | 说明 | +|---|---|---| +| `OPENCODE_API_KEY` | 无 | 逗号分隔的 API key 列表 | +| `OPENCODE_THROTTLE_ENABLE` | `true` | 设为 `false` 可禁用跨进程限流记录 | +| `OPENCODE_THROTTLE_DURATION` | `120`（分钟） | 限流记录的有效期 | +| `OPENCODE_WELAN_LOG` | `true` | 设为 `false` 可禁用 welan.txt 日志 | + +### 跨进程限流（多个 opencode 进程共存时） + +限流状态保存在 `~/.config/opencode/throttle.json`。当某个 key 触发 429 时，当前进程把该 key 写入此文件并附上失效时间。其他进程启动时读取该文件，自动跳过仍在限流期内的 key——无需任何手动操作。 + +``` +~/.config/opencode/ +├── throttle.json # 跨进程限流状态（自动管理） +└── welan-log.txt # key 轮转决策日志 +``` + +### 日志（welan.txt） + +key 轮转决策会写两份日志： + +- `~/.config/opencode/welan-log.txt`：专用排查日志，保留进程前缀。 +- `~/.local/state/opencode/log/opencode.log`（随 `Global.Path.log` 配置变化）：遵循 opencode 原有结构化日志格式。设置 `OPENCODE_PRINT_LOGS=1` 时，也按原有日志机制输出到 stderr。 + +`welan-log.txt` 示例： + +``` +[012300000Z] [2026-07-10T09:23:00.000] [INFO] key-rotation: start, 2 key(s) configured +[012300000Z] [2026-07-10T09:23:05.000] [WARN] key-rotation: key ***key-1 throttled for 120 minutes, writing throttle record +[012300000Z] [2026-07-10T09:23:05.000] [WARN] key-rotation: key ***key-1 quota_limit, trying next +[012300000Z] [2026-07-10T09:23:05.000] [INFO] key-rotation: attempt 2 with key ***key-2 +[012300000Z] [2026-07-10T09:23:10.000] [INFO] key-rotation: success with key ***key-2 +``` + +Key 在日志中脱敏，只显示最后 6 位。第二列（如 `[012300000Z]`）是进程级前缀，同一 CLI 进程的所有日志共享该值，方便在多进程并发时区分来源。 + +`opencode.log` 示例： + +``` +timestamp=2026-07-10T09:23:05.000Z level=WARN run=abcd1234 message="key-rotation: key ***key-1 quota_limit, trying next" +``` + +--- + +## 内部实现 + +### 模块结构 + +``` +packages/opencode/src/ +├── provider/ +│ ├── throttle-store.ts # 读写 throttle.json，Flock 写锁 +│ ├── rotation-logger.ts # 追加写 welan-log.txt，fire-and-forget +│ └── key-rotator.ts # 解析 key 列表，管理轮转状态 +├── session/ +│ └── retry.ts # isInvalidKeyAPIError（401 检测） +└── cli/cmd/ + └── run.ts # runWithKeyRotation 轮转主循环 +``` + +这三个新模块没有任何 Effect / Provider / Session 依赖，可以单独使用。 + +### 轮转流程 + +``` +opencode run "prompt" + │ + ▼ +runWithKeyRotation(createSdk) + │ + ├─ KeyRotator.selectKey() + │ ├─ 读 throttle.json（无锁） + │ ├─ 跳过限流期内的 key + │ └─ 跳过本进程已标记无效的 key + │ + ├─ process.env.OPENCODE_API_KEY = selectedKey + ├─ disposeInstance(directory) ← 仅在第 2 次以后调用 + ├─ Server.Default.reset() ← 仅在第 2 次以后调用 + │ └─ 清除目录级 InstanceState 与惰性 Server，下次 fetch 读新 key + │ + ├─ execute(sdk, overrideSessionID) ← 透传上一次的 sessionID + │ └─ 遇到可轮换错误时抛出 KeyRotationRetry + │ + ├─ quota_limit → KeyRotator.recordThrottle(key) → 写 throttle.json → 换 key + ├─ invalid_key → KeyRotator.markInvalid(key) → 进程内跳过 → 换 key + └─ success → 结束 +``` + +### 关键设计决策 + +**disposeInstance() + Server.Default.reset()** — Provider key 缓存在目录级 `InstanceState` 中。换 key 前先清除当前目录的 InstanceState，再 reset 惰性 Server，下一次 HTTP 请求会用新的 `process.env.OPENCODE_API_KEY` 重建本地执行路径。这样不需要在 Provider/Env 层监听全局 env 变化。 + +**KeyRotationRetry** — `execute()` 保留原来的成功/失败返回语义。只有 429/quota/rate-limit 和 401 这两类可轮换错误会抛出 `KeyRotationRetry`，由外层 `runWithKeyRotation()` 捕获并换 key。 + +**Session ID 透传** — `overrideSessionID` 在轮转后传给下一次 `execute()`，使新 key 的请求继续使用同一个 SQLite session，保留上下文。 + +**锁策略** — `isThrottled`（读）不加锁，宁可偶发读到旧数据也不阻塞 API 调用。`addThrottle` / `cleanExpired`（写）使用 `Flock.acquire`，超时 2s 后放弃写入（宁漏记，不阻塞）。进程崩溃导致的僵尸锁通过 `staleMs: 10_000` 自动清理。 + +**throttleEnabled 默认开启** — `OPENCODE_THROTTLE_ENABLE !== "false"`，未设置时视为开启。`retry.ts` 中的同名判断也用同一逻辑，确保 429 能直接冒泡到外层轮转循环，而不是在 SDK 内部重试同一个已限流的 key。 diff --git a/docs/superpowers/plans/2026-07-09-key-rotation-throttle.md b/docs/superpowers/plans/2026-07-09-key-rotation-throttle.md new file mode 100644 index 000000000000..862c2e429ae5 --- /dev/null +++ b/docs/superpowers/plans/2026-07-09-key-rotation-throttle.md @@ -0,0 +1,1212 @@ +# Key Rotation & Throttle Suppression Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add cross-process throttle suppression and in-process multi-key fallback for `OPENCODE_API_KEY`, writing decisions to `welan.txt` in addition to the existing log channel. + +**Architecture:** Three new standalone modules (`throttle-store.ts`, `welan-logger.ts`, `key-rotator.ts`) with zero Effect/Provider dependencies. Integration into `run.ts` via a `runWithKeyRotation` loop that calls `Server.Default.reset()` between key attempts — the lazy singleton re-initializes with the updated env var on the next request, no Effect re-run needed. + +**Tech Stack:** Bun, TypeScript, Node.js `fs/promises`, existing `Flock` from `@opencode-ai/core/util/flock`, existing `Global.Path` from `@opencode-ai/core/global`. + +## Global Constraints + +- All new source files: `packages/opencode/src/provider/` +- All new test files: `packages/opencode/test/provider/` or `packages/opencode/test/cli/run/` +- Run tests from `packages/opencode/` directory (never repo root) +- Test command: `bun test ` from `packages/opencode/` +- Typecheck: `bun typecheck` from `packages/opencode/` +- Commits: `git commit -s -S -m "type(scope): summary"` +- No imports of Effect, Provider, Session, or Auth in new modules +- Key masking in logs: last 6 chars only, prefix `***` +- `OPENCODE_API_KEY` split on `,`, trim, drop empty strings +- `OPENCODE_THROTTLE_ENABLE` default `"true"` (disabled only when `=== "false"`) +- `OPENCODE_THROTTLE_DURATION` default `120` minutes +- `OPENCODE_WELAN_LOG` default `"true"` (disabled only when `=== "false"`) +- throttle.json path: `path.join(Global.Path.config, "throttle.json")` +- welan.txt path: `path.join(Global.Path.config, "welan.txt")` +- Flock key for throttle: `"throttle-store"`, `dir: Global.Path.config`, `timeoutMs: 2_000`, `staleMs: 10_000` + +--- + +## File Map + +| File | Action | Responsibility | +|---|---|---| +| `src/provider/throttle-store.ts` | Create | Read/write throttle.json; Flock-based write locking | +| `src/provider/welan-logger.ts` | Create | Append to welan.txt, fire-and-forget | +| `src/provider/key-rotator.ts` | Create | Parse key list, select next available key | +| `src/session/retry.ts` | Modify | Add `isInvalidKeyAPIError` (~5 lines) | +| `src/cli/cmd/run.ts` | Modify | `execute()` returns `AttemptResult`; add `runWithKeyRotation` | +| `test/provider/throttle-store.test.ts` | Create | Unit tests for ThrottleStore | +| `test/provider/welan-logger.test.ts` | Create | Unit tests for WelANLogger | +| `test/provider/key-rotator.test.ts` | Create | Unit tests for KeyRotator | +| `test/cli/run/run-key-rotation.test.ts` | Create | E2E tests via cliIt.live | + +--- + +## Task 1: ThrottleStore + +**Files:** +- Create: `packages/opencode/src/provider/throttle-store.ts` +- Create: `packages/opencode/test/provider/throttle-store.test.ts` + +**Interfaces:** +- Produces: + ```typescript + // throttle.json record + type ThrottleRecord = { + source: string // env var name, e.g. "OPENCODE_API_KEY" + key: string // full API key + startTime: number // Unix ms + endTime: number // Unix ms + } + + function isThrottled(source: string, key: string): Promise + function addThrottle(source: string, key: string, durationMinutes: number): Promise + function cleanExpired(source: string, key: string): Promise + export * as ThrottleStore from "./throttle-store" + ``` + +- [ ] **Step 1: Write the failing tests** + + Create `packages/opencode/test/provider/throttle-store.test.ts`: + + ```typescript + import { afterEach, describe, expect, it } from "bun:test" + import path from "path" + import fs from "fs/promises" + import os from "os" + + // Override Global.Path.config to a temp dir for tests + const tmpDir = path.join(os.tmpdir(), "opencode-throttle-test-" + process.pid) + const throttleFile = path.join(tmpDir, "throttle.json") + + // Patch the global path before importing the module under test + process.env.XDG_CONFIG_HOME = path.join(tmpDir, "xdg") + + // Dynamic import so env patch applies first + const { ThrottleStore } = await import("../../src/provider/throttle-store") + + afterEach(async () => { + await fs.rm(tmpDir, { recursive: true, force: true }) + await fs.mkdir(tmpDir, { recursive: true }) + }) + + describe("ThrottleStore.isThrottled", () => { + it("returns false when throttle.json does not exist", async () => { + expect(await ThrottleStore.isThrottled("OPENCODE_API_KEY", "key1")).toBe(false) + }) + + it("returns true when key has an active throttle record", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([{ source: "OPENCODE_API_KEY", key: "key1", startTime: now - 1000, endTime: now + 60_000 }]), + ) + expect(await ThrottleStore.isThrottled("OPENCODE_API_KEY", "key1")).toBe(true) + }) + + it("returns false when key record is expired", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([{ source: "OPENCODE_API_KEY", key: "key1", startTime: now - 120_000, endTime: now - 1000 }]), + ) + expect(await ThrottleStore.isThrottled("OPENCODE_API_KEY", "key1")).toBe(false) + }) + + it("returns false for a different key not in the file", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([{ source: "OPENCODE_API_KEY", key: "key1", startTime: now - 1000, endTime: now + 60_000 }]), + ) + expect(await ThrottleStore.isThrottled("OPENCODE_API_KEY", "key2")).toBe(false) + }) + + it("returns false on JSON parse error (file is corrupt)", async () => { + await fs.writeFile(throttleFile, "not-json") + expect(await ThrottleStore.isThrottled("OPENCODE_API_KEY", "key1")).toBe(false) + }) + }) + + describe("ThrottleStore.addThrottle", () => { + it("creates throttle.json with a new record", async () => { + await ThrottleStore.addThrottle("OPENCODE_API_KEY", "key1", 120) + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(1) + expect(data[0].source).toBe("OPENCODE_API_KEY") + expect(data[0].key).toBe("key1") + expect(data[0].endTime - data[0].startTime).toBe(120 * 60 * 1000) + }) + + it("upserts: updates existing record for same key", async () => { + await ThrottleStore.addThrottle("OPENCODE_API_KEY", "key1", 60) + await ThrottleStore.addThrottle("OPENCODE_API_KEY", "key1", 120) + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(1) + expect(data[0].endTime - data[0].startTime).toBe(120 * 60 * 1000) + }) + + it("appends: keeps other keys when adding a new one", async () => { + await ThrottleStore.addThrottle("OPENCODE_API_KEY", "key1", 60) + await ThrottleStore.addThrottle("OPENCODE_API_KEY", "key2", 120) + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(2) + }) + }) + + describe("ThrottleStore.cleanExpired", () => { + it("removes expired record for the given key", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([ + { source: "OPENCODE_API_KEY", key: "key1", startTime: now - 120_000, endTime: now - 1000 }, + { source: "OPENCODE_API_KEY", key: "key2", startTime: now - 1000, endTime: now + 60_000 }, + ]), + ) + await ThrottleStore.cleanExpired("OPENCODE_API_KEY", "key1") + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(1) + expect(data[0].key).toBe("key2") + }) + + it("does nothing when key is not in the file", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([{ source: "OPENCODE_API_KEY", key: "key2", startTime: now, endTime: now + 60_000 }]), + ) + await ThrottleStore.cleanExpired("OPENCODE_API_KEY", "key1") + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(1) + }) + }) + ``` + +- [ ] **Step 2: Run test to verify it fails** + + ```bash + cd packages/opencode + bun test test/provider/throttle-store.test.ts 2>&1 | head -20 + ``` + + Expected: import error — module does not exist yet. + +- [ ] **Step 3: Implement ThrottleStore** + + Create `packages/opencode/src/provider/throttle-store.ts`: + + ```typescript + import path from "path" + import fs from "fs/promises" + import { Flock } from "@opencode-ai/core/util/flock" + import { Global } from "@opencode-ai/core/global" + + type ThrottleRecord = { + source: string + key: string + startTime: number + endTime: number + } + + const filePath = () => path.join(Global.Path.config, "throttle.json") + + async function readRecords(): Promise { + try { + const raw = await fs.readFile(filePath(), "utf8") + const parsed = JSON.parse(raw) + if (!Array.isArray(parsed)) return [] + return parsed as ThrottleRecord[] + } catch { + return [] + } + } + + async function writeRecords(records: ThrottleRecord[]): Promise { + await fs.mkdir(path.dirname(filePath()), { recursive: true }) + await fs.writeFile(filePath(), JSON.stringify(records, null, 2)) + } + + export async function isThrottled(source: string, key: string): Promise { + const records = await readRecords() + const now = Date.now() + const record = records.find((r) => r.source === source && r.key === key) + if (!record) return false + if (now < record.endTime) return true + // Expired: clean up in the background, do not block + void cleanExpired(source, key) + return false + } + + export async function addThrottle(source: string, key: string, durationMinutes: number): Promise { + let lease: Awaited> | null = null + try { + lease = await Flock.acquire("throttle-store", { + dir: Global.Path.config, + timeoutMs: 2_000, + staleMs: 10_000, + }) + } catch { + // Timed out waiting for lock: prefer missing the write over blocking the caller + return + } + try { + const records = await readRecords() + const now = Date.now() + const endTime = now + durationMinutes * 60 * 1000 + const idx = records.findIndex((r) => r.source === source && r.key === key) + const record: ThrottleRecord = { source, key, startTime: now, endTime } + if (idx >= 0) records[idx] = record + else records.push(record) + await writeRecords(records) + } finally { + await lease.release() + } + } + + export async function cleanExpired(source: string, key: string): Promise { + let lease: Awaited> | null = null + try { + lease = await Flock.acquire("throttle-store", { + dir: Global.Path.config, + timeoutMs: 2_000, + staleMs: 10_000, + }) + } catch { + return + } + try { + const records = await readRecords() + const now = Date.now() + const filtered = records.filter((r) => !(r.source === source && r.key === key && now >= r.endTime)) + await writeRecords(filtered) + } finally { + await lease.release() + } + } + + export * as ThrottleStore from "./throttle-store" + ``` + +- [ ] **Step 4: Run tests to verify they pass** + + ```bash + cd packages/opencode + bun test test/provider/throttle-store.test.ts + ``` + + Expected: all tests pass. + +- [ ] **Step 5: Typecheck** + + ```bash + cd packages/opencode + bun typecheck + ``` + + Expected: no errors. + +- [ ] **Step 6: Commit** + + ```bash + cd packages/opencode + git add src/provider/throttle-store.ts test/provider/throttle-store.test.ts + git commit -s -S -m "feat(opencode): add ThrottleStore for cross-process key throttle suppression" + ``` + +--- + +## Task 2: WelANLogger + +**Files:** +- Create: `packages/opencode/src/provider/welan-logger.ts` +- Create: `packages/opencode/test/provider/welan-logger.test.ts` + +**Interfaces:** +- Produces: + ```typescript + type Level = "info" | "warn" | "error" + function log(level: Level, message: string): void // fire-and-forget + export * as WelANLogger from "./welan-logger" + ``` + +- [ ] **Step 1: Write the failing test** + + Create `packages/opencode/test/provider/welan-logger.test.ts`: + + ```typescript + import { afterEach, describe, expect, it } from "bun:test" + import path from "path" + import fs from "fs/promises" + import os from "os" + + const tmpDir = path.join(os.tmpdir(), "opencode-welan-test-" + process.pid) + process.env.XDG_CONFIG_HOME = path.join(tmpDir, "xdg") + + const { WelANLogger } = await import("../../src/provider/welan-logger") + + const welanFile = path.join(tmpDir, "welan.txt") + + afterEach(async () => { + await fs.rm(tmpDir, { recursive: true, force: true }) + await fs.mkdir(tmpDir, { recursive: true }) + delete process.env.OPENCODE_WELAN_LOG + }) + + function sleep(ms: number) { + return new Promise((r) => setTimeout(r, ms)) + } + + describe("WelANLogger.log", () => { + it("appends a line to welan.txt with correct format", async () => { + WelANLogger.log("info", "test message") + await sleep(50) // fire-and-forget: wait for write + const content = await fs.readFile(welanFile, "utf8") + expect(content).toMatch(/\[.*\] \[INFO\] test message\n/) + }) + + it("appends multiple lines in order", async () => { + WelANLogger.log("info", "first") + WelANLogger.log("warn", "second") + await sleep(50) + const content = await fs.readFile(welanFile, "utf8") + const lines = content.trim().split("\n") + expect(lines).toHaveLength(2) + expect(lines[0]).toContain("[INFO] first") + expect(lines[1]).toContain("[WARN] second") + }) + + it("does nothing when OPENCODE_WELAN_LOG=false", async () => { + process.env.OPENCODE_WELAN_LOG = "false" + WelANLogger.log("info", "should not appear") + await sleep(50) + const exists = await fs.stat(welanFile).then(() => true).catch(() => false) + expect(exists).toBe(false) + }) + }) + ``` + +- [ ] **Step 2: Run test to verify it fails** + + ```bash + cd packages/opencode + bun test test/provider/welan-logger.test.ts 2>&1 | head -10 + ``` + + Expected: import error. + +- [ ] **Step 3: Implement WelANLogger** + + Create `packages/opencode/src/provider/welan-logger.ts`: + + ```typescript + import path from "path" + import fs from "fs/promises" + import { Global } from "@opencode-ai/core/global" + + export type Level = "info" | "warn" | "error" + + const filePath = () => path.join(Global.Path.config, "welan.txt") + + export function log(level: Level, message: string): void { + if (process.env.OPENCODE_WELAN_LOG === "false") return + const timestamp = new Date().toISOString() + const line = `[${timestamp}] [${level.toUpperCase()}] ${message}\n` + // Fire-and-forget: never block the caller + void fs + .mkdir(path.dirname(filePath()), { recursive: true }) + .then(() => fs.appendFile(filePath(), line)) + .catch(() => {}) + } + + export * as WelANLogger from "./welan-logger" + ``` + +- [ ] **Step 4: Run tests to verify they pass** + + ```bash + cd packages/opencode + bun test test/provider/welan-logger.test.ts + ``` + + Expected: all tests pass. + +- [ ] **Step 5: Typecheck and commit** + + ```bash + cd packages/opencode + bun typecheck + git add src/provider/welan-logger.ts test/provider/welan-logger.test.ts + git commit -s -S -m "feat(opencode): add WelANLogger for supplemental welan.txt logging" + ``` + +--- + +## Task 3: KeyRotator + +**Files:** +- Create: `packages/opencode/src/provider/key-rotator.ts` +- Create: `packages/opencode/test/provider/key-rotator.test.ts` + +**Interfaces:** +- Consumes: `ThrottleStore.isThrottled`, `ThrottleStore.addThrottle`, `WelANLogger.log` +- Produces: + ```typescript + function parseKeys(): string[] + + class KeyRotator { + constructor(keys: string[]) + async selectKey(): Promise + async recordThrottle(key: string): Promise + markInvalid(key: string): void + hasAlternative(currentKey: string): boolean + } + + export * as KeyRotator from "./key-rotator" + ``` + +- [ ] **Step 1: Write the failing tests** + + Create `packages/opencode/test/provider/key-rotator.test.ts`: + + ```typescript + import { afterEach, describe, expect, it, mock } from "bun:test" + + // Mock ThrottleStore before importing KeyRotator + const mockIsThrottled = mock(async (_source: string, _key: string) => false) + const mockAddThrottle = mock(async (_source: string, _key: string, _dur: number) => {}) + + mock.module("../../src/provider/throttle-store", () => ({ + ThrottleStore: { isThrottled: mockIsThrottled, addThrottle: mockAddThrottle, cleanExpired: async () => {} }, + })) + + const { KeyRotator, parseKeys } = await import("../../src/provider/key-rotator") + + afterEach(() => { + mockIsThrottled.mockReset() + mockAddThrottle.mockReset() + mockIsThrottled.mockImplementation(async () => false) + delete process.env.OPENCODE_API_KEY + delete process.env.OPENCODE_THROTTLE_ENABLE + delete process.env.OPENCODE_THROTTLE_DURATION + }) + + describe("parseKeys", () => { + it("returns empty array when env var not set", () => { + delete process.env.OPENCODE_API_KEY + expect(parseKeys()).toEqual([]) + }) + + it("returns single key", () => { + process.env.OPENCODE_API_KEY = "sk-abc" + expect(parseKeys()).toEqual(["sk-abc"]) + }) + + it("returns multiple keys split by comma", () => { + process.env.OPENCODE_API_KEY = "sk-abc,sk-def,sk-ghi" + expect(parseKeys()).toEqual(["sk-abc", "sk-def", "sk-ghi"]) + }) + + it("trims whitespace around commas", () => { + process.env.OPENCODE_API_KEY = "sk-abc , sk-def" + expect(parseKeys()).toEqual(["sk-abc", "sk-def"]) + }) + + it("drops empty strings after split", () => { + process.env.OPENCODE_API_KEY = "sk-abc,,sk-def" + expect(parseKeys()).toEqual(["sk-abc", "sk-def"]) + }) + }) + + describe("KeyRotator.selectKey", () => { + it("returns first key when none are throttled or invalid", async () => { + const r = new KeyRotator(["key1", "key2"]) + expect(await r.selectKey()).toBe("key1") + }) + + it("skips throttled key and returns next", async () => { + mockIsThrottled.mockImplementation(async (_s, key) => key === "key1") + const r = new KeyRotator(["key1", "key2"]) + expect(await r.selectKey()).toBe("key2") + }) + + it("skips invalid key and returns next", async () => { + const r = new KeyRotator(["key1", "key2"]) + r.markInvalid("key1") + expect(await r.selectKey()).toBe("key2") + }) + + it("returns null when all keys are throttled", async () => { + mockIsThrottled.mockImplementation(async () => true) + const r = new KeyRotator(["key1", "key2"]) + expect(await r.selectKey()).toBeNull() + }) + + it("returns null when all keys are invalid", async () => { + const r = new KeyRotator(["key1", "key2"]) + r.markInvalid("key1") + r.markInvalid("key2") + expect(await r.selectKey()).toBeNull() + }) + + it("skips throttle check when OPENCODE_THROTTLE_ENABLE=false", async () => { + process.env.OPENCODE_THROTTLE_ENABLE = "false" + mockIsThrottled.mockImplementation(async () => true) // would throttle if checked + const r = new KeyRotator(["key1"]) + expect(await r.selectKey()).toBe("key1") // not skipped because throttle disabled + expect(mockIsThrottled).not.toHaveBeenCalled() + }) + }) + + describe("KeyRotator.recordThrottle", () => { + it("calls ThrottleStore.addThrottle with correct source and default duration", async () => { + const r = new KeyRotator(["key1"]) + await r.recordThrottle("key1") + expect(mockAddThrottle).toHaveBeenCalledWith("OPENCODE_API_KEY", "key1", 120) + }) + + it("uses OPENCODE_THROTTLE_DURATION when set", async () => { + process.env.OPENCODE_THROTTLE_DURATION = "60" + const r = new KeyRotator(["key1"]) + await r.recordThrottle("key1") + expect(mockAddThrottle).toHaveBeenCalledWith("OPENCODE_API_KEY", "key1", 60) + }) + }) + + describe("KeyRotator.markInvalid", () => { + it("causes the key to be skipped in selectKey", async () => { + const r = new KeyRotator(["key1", "key2"]) + r.markInvalid("key1") + expect(await r.selectKey()).toBe("key2") + }) + + it("does not write to throttle.json", async () => { + const r = new KeyRotator(["key1"]) + r.markInvalid("key1") + expect(mockAddThrottle).not.toHaveBeenCalled() + }) + }) + + describe("KeyRotator.hasAlternative", () => { + it("returns true when other non-invalid keys exist", () => { + const r = new KeyRotator(["key1", "key2", "key3"]) + expect(r.hasAlternative("key1")).toBe(true) + }) + + it("returns false when only one key and it is current", () => { + const r = new KeyRotator(["key1"]) + expect(r.hasAlternative("key1")).toBe(false) + }) + + it("returns false when other keys are all invalid", () => { + const r = new KeyRotator(["key1", "key2"]) + r.markInvalid("key2") + expect(r.hasAlternative("key1")).toBe(false) + }) + }) + ``` + +- [ ] **Step 2: Run test to verify it fails** + + ```bash + cd packages/opencode + bun test test/provider/key-rotator.test.ts 2>&1 | head -10 + ``` + + Expected: import error. + +- [ ] **Step 3: Implement KeyRotator** + + Create `packages/opencode/src/provider/key-rotator.ts`: + + ```typescript + import { ThrottleStore } from "./throttle-store" + import { WelANLogger } from "./welan-logger" + + const SOURCE = "OPENCODE_API_KEY" + + export function parseKeys(): string[] { + const raw = process.env.OPENCODE_API_KEY ?? "" + return raw + .split(",") + .map((k) => k.trim()) + .filter(Boolean) + } + + function throttleDuration(): number { + const raw = process.env.OPENCODE_THROTTLE_DURATION + const parsed = raw ? parseInt(raw, 10) : NaN + return Number.isFinite(parsed) && parsed > 0 ? parsed : 120 + } + + function isThrottleEnabled(): boolean { + return process.env.OPENCODE_THROTTLE_ENABLE !== "false" + } + + function maskKey(key: string): string { + return `***${key.slice(-6)}` + } + + export class KeyRotator { + private readonly invalid = new Set() + + constructor(private readonly keys: string[]) {} + + async selectKey(): Promise { + for (const key of this.keys) { + if (this.invalid.has(key)) continue + if (isThrottleEnabled() && (await ThrottleStore.isThrottled(SOURCE, key))) { + WelANLogger.log("info", `key-rotation: key ${maskKey(key)} is throttled, skipping`) + continue + } + return key + } + return null + } + + async recordThrottle(key: string): Promise { + const duration = throttleDuration() + WelANLogger.log( + "warn", + `key-rotation: key ${maskKey(key)} throttled for ${duration} minutes, writing throttle record`, + ) + await ThrottleStore.addThrottle(SOURCE, key, duration) + } + + markInvalid(key: string): void { + WelANLogger.log("warn", `key-rotation: key ${maskKey(key)} marked invalid (401), skipping for this process`) + this.invalid.add(key) + } + + hasAlternative(currentKey: string): boolean { + return this.keys.some((k) => k !== currentKey && !this.invalid.has(k)) + } + } + + export * as KeyRotator from "./key-rotator" + ``` + +- [ ] **Step 4: Run tests to verify they pass** + + ```bash + cd packages/opencode + bun test test/provider/key-rotator.test.ts + ``` + + Expected: all tests pass. + +- [ ] **Step 5: Typecheck and commit** + + ```bash + cd packages/opencode + bun typecheck + git add src/provider/key-rotator.ts test/provider/key-rotator.test.ts + git commit -s -S -m "feat(opencode): add KeyRotator for OPENCODE_API_KEY multi-key rotation" + ``` + +--- + +## Task 4: retry.ts — add isInvalidKeyAPIError + +**Files:** +- Modify: `packages/opencode/src/session/retry.ts` (add ~5 lines after `isQuotaOrRateLimitAPIError`) + +**Interfaces:** +- Consumes: existing `isRecord` from `@/util/record` +- Produces: + ```typescript + export function isInvalidKeyAPIError(error: unknown): boolean + // → true if error.name === "APIError" && error.data.statusCode === 401 + ``` + +- [ ] **Step 1: Add the function** + + In `packages/opencode/src/session/retry.ts`, after the `isQuotaOrRateLimitAPIError` function (currently ending at line 62), add: + + ```typescript + export function isInvalidKeyAPIError(error: unknown): boolean { + if (!isRecord(error) || error.name !== "APIError" || !isRecord(error.data)) return false + return error.data.statusCode === 401 + } + ``` + +- [ ] **Step 2: Verify the export is re-exported** + + The file already ends with `export * as SessionRetry from "./retry"` — no change needed there. + +- [ ] **Step 3: Typecheck** + + ```bash + cd packages/opencode + bun typecheck + ``` + + Expected: no errors. + +- [ ] **Step 4: Commit** + + ```bash + cd packages/opencode + git add src/session/retry.ts + git commit -s -S -m "feat(opencode): add isInvalidKeyAPIError to retry helpers" + ``` + +--- + +## Task 5: run.ts — AttemptResult + runWithKeyRotation + +**Files:** +- Modify: `packages/opencode/src/cli/cmd/run.ts` + +**Interfaces:** +- Consumes: `KeyRotator` (parse/class), `WelANLogger.log`, `SessionRetry.isInvalidKeyAPIError`, `Server.Default.reset` from `@/server/server` +- The `execute()` function internal to the `Effect.promise` callback is modified to return `AttemptResult` instead of `string | undefined`. + +**Context for this task:** In `run.ts`, the non-interactive, non-attach code path (around line 954) creates an embedded server via `Server.Default()` inside a lazy `fetchFn`, creates an SDK, then calls `execute(sdk)`. The `execute` function (line 671) creates or retrieves a session, runs the event loop, and currently returns `string | undefined` (error or undefined). `Server.Default` is a lazy singleton; calling `Server.Default.reset()` clears the cached instance so the next HTTP request creates a fresh server — which re-reads `process.env.OPENCODE_API_KEY`. + +- [ ] **Step 1: Add import for KeyRotator and WelANLogger** + + At the top of `packages/opencode/src/cli/cmd/run.ts`, after the existing imports, add: + + ```typescript + import { KeyRotator, parseKeys } from "@/provider/key-rotator" + import { WelANLogger } from "@/provider/welan-logger" + ``` + + Also add `isInvalidKeyAPIError` to the existing SessionRetry import: + + ```typescript + // existing line (around line 28): + import { SessionRetry } from "@/session/retry" + // SessionRetry now also exports isInvalidKeyAPIError via the namespace re-export + ``` + + No change needed to the SessionRetry import — `isInvalidKeyAPIError` is already re-exported via `export * as SessionRetry from "./retry"`. + +- [ ] **Step 2: Add AttemptResult type** + + After the existing `type ModelInput = ...` line (around line 30), add: + + ```typescript + type AttemptResult = { + sessionID: string | undefined + exitReason: "success" | "quota_limit" | "invalid_key" | "error" + error?: string + } + ``` + +- [ ] **Step 3: Modify execute() to return AttemptResult** + + The `execute` function (line 671) currently returns `Promise` implicitly. Modify: + + a. Change function signature comment: the function now returns `Promise`. + + b. At the top of `execute`, add a variable to track the session ID: + ```typescript + async function execute(sdk: OpencodeClient, overrideSessionID?: string): Promise { + // If key rotation provided a session ID from a prior attempt, pass it via args.session override + const sess = await session(sdk, overrideSessionID) + ``` + + Modify the `session` helper to accept an optional override: + ```typescript + async function session(sdk: OpencodeClient, overrideID?: string): Promise { + const id = overrideID ?? args.session + if (id) { + // ... existing args.session branch but using `id` instead of `args.session` + ``` + + c. Replace every `return error` inside `loop()` and `execute()` with a structured return: + - Rate-limit return: `return { sessionID, exitReason: "quota_limit", error }` + - Invalid key return: `return { sessionID, exitReason: "invalid_key", error }` + - Normal end (after `await finish()`): `return { sessionID, exitReason: "success" }` + - Error exit: `return { sessionID, exitReason: "error", error }` + + Specifically, in `loop()`: + + ```typescript + // Replace (line ~785-791): + const limit = SessionRetry.isQuotaOrRateLimitAPIError(props.error) + if (emit("error", { error: props.error })) { + if (limit) return error + continue + } + UI.error(err) + if (limit) return error + + // With: + const isQuota = SessionRetry.isQuotaOrRateLimitAPIError(props.error) + const isInvalid = SessionRetry.isInvalidKeyAPIError(props.error) + if (emit("error", { error: props.error })) { + if (isQuota) return { exitReason: "quota_limit", error } + if (isInvalid) return { exitReason: "invalid_key", error } + continue + } + UI.error(err) + if (isQuota) return { exitReason: "quota_limit", error } + if (isInvalid) return { exitReason: "invalid_key", error } + ``` + + And for the retry-status block (line ~796-800): + ```typescript + // Replace: + if (status.type === "retry" && SessionRetry.isQuotaOrRateLimitRetryStatus(status)) { + error = error ? error + EOL + status.message : status.message + if (emit("error", { error: status })) return error + UI.error(status.message) + return error + } + + // With: + if (status.type === "retry" && SessionRetry.isQuotaOrRateLimitRetryStatus(status)) { + error = error ? error + EOL + status.message : status.message + if (emit("error", { error: status })) return { sessionID, exitReason: "quota_limit", error } + UI.error(status.message) + return { sessionID, exitReason: "quota_limit", error } + } + ``` + + `loop()` now returns `Promise<{ exitReason: "quota_limit" | "invalid_key"; error: string } | undefined>` instead of `Promise`. The `execute()` caller of `loop()` updates accordingly. + + The final success path in `execute()`: + ```typescript + await finish() + return { sessionID, exitReason: "success" } + ``` + +- [ ] **Step 4: Add runWithKeyRotation and wire up the non-interactive path** + + Add this function inside the `Effect.promise(async () => { ... })` block, after the existing helper functions (`session`, `share`, `execute`, etc.) but before the final path branches. In `run.ts`, the non-interactive non-attach branch (currently ending with `await execute(sdk)` around line 967) is replaced: + + ```typescript + // New function inside the async block: + async function runWithKeyRotation(createSdk: () => OpencodeClient): Promise { + const { Server } = await import("@/server/server") + const keys = parseKeys() + const rotator = new KeyRotator(keys) + let currentSessionID: string | undefined = args.session + + WelANLogger.log("info", `key-rotation: start, ${keys.length} key(s) configured`) + + let attempt = 0 + while (true) { + const key = await rotator.selectKey() + if (!key) { + WelANLogger.log("error", "key-rotation: all OPENCODE_API_KEY keys exhausted or throttled") + process.exitCode = 1 + return + } + + attempt++ + process.env.OPENCODE_API_KEY = key + if (attempt > 1) Server.Default.reset() // force fresh server with new key + WelANLogger.log("info", `key-rotation: attempt ${attempt} with key ***${key.slice(-6)}`) + + const result = await execute(createSdk(), currentSessionID) + currentSessionID = result.sessionID ?? currentSessionID + + if (result.exitReason === "quota_limit") { + await rotator.recordThrottle(key) + WelANLogger.log( + "warn", + `key-rotation: key ***${key.slice(-6)} quota_limit, ${rotator.hasAlternative(key) ? "trying next" : "no more keys"}`, + ) + if (!rotator.hasAlternative(key)) { + if (result.error) process.exitCode = 1 + return + } + continue + } + + if (result.exitReason === "invalid_key") { + rotator.markInvalid(key) + WelANLogger.log( + "warn", + `key-rotation: key ***${key.slice(-6)} invalid, ${rotator.hasAlternative(key) ? "trying next" : "no more keys"}`, + ) + if (!rotator.hasAlternative(key)) { + if (result.error) process.exitCode = 1 + return + } + continue + } + + if (result.exitReason === "error" || result.error) { + process.exitCode = 1 + } + WelANLogger.log("info", `key-rotation: ${result.exitReason} with key ***${key.slice(-6)}`) + return + } + } + ``` + + Replace the existing non-interactive non-attach tail (lines ~954-967): + + ```typescript + // Before (existing): + const fetchFn = (async (input, init?) => { + const { Server } = await import("@/server/server") + ... + return Server.Default().app.fetch(...) + }) as typeof globalThis.fetch + const sdk = createOpencodeClient({ baseUrl: "http://opencode.internal", fetch: fetchFn, directory }) + await execute(sdk) + + // After: + const fetchFn = (async (input: RequestInfo | URL, init?: RequestInit) => { + const { Server } = await import("@/server/server") + const request = new Request(input, init) + const headers = new Headers(request.headers) + const auth = ServerAuth.header() + if (auth) headers.set("Authorization", auth) + return Server.Default().app.fetch(new Request(request, { headers })) + }) as typeof globalThis.fetch + + const createSdk = () => + createOpencodeClient({ baseUrl: "http://opencode.internal", fetch: fetchFn, directory }) + + await runWithKeyRotation(createSdk) + ``` + +- [ ] **Step 5: Run gofmt equivalent (TypeScript formatter)** + + ```bash + cd packages/opencode + bun typecheck + ``` + + Expected: no errors. + +- [ ] **Step 6: Commit** + + ```bash + cd packages/opencode + git add src/cli/cmd/run.ts src/session/retry.ts + git commit -s -S -m "feat(opencode): integrate key rotation loop into run command" + ``` + +--- + +## Task 6: E2E Tests + +**Files:** +- Create: `packages/opencode/test/cli/run/run-key-rotation.test.ts` + +**Context:** These tests follow the pattern in `test/cli/run/run-process-limit.test.ts`. `cliIt.live` spawns a real `opencode run` subprocess against a mock LLM server (`llm`). The test sets `OPENCODE_API_KEY` to comma-separated keys and controls which one the mock server accepts. + +- [ ] **Step 1: Write the E2E tests** + + Create `packages/opencode/test/cli/run/run-key-rotation.test.ts`: + + ```typescript + // Key rotation behavior for `opencode run` (non-interactive subprocess). + // Kept in a separate file to avoid competing with the main CLI regression suite. + import { afterEach, beforeEach, describe, expect } from "bun:test" + import path from "path" + import fs from "fs/promises" + import os from "os" + import { Effect } from "effect" + import { cliIt } from "../../lib/cli-process" + + const tmpDir = path.join(os.tmpdir(), "opencode-key-rotation-e2e-" + process.pid) + const throttleFile = path.join(tmpDir, "throttle.json") + + beforeEach(async () => { + await fs.mkdir(tmpDir, { recursive: true }) + process.env.XDG_CONFIG_HOME = path.join(tmpDir, "xdg") + }) + + afterEach(async () => { + await fs.rm(tmpDir, { recursive: true, force: true }) + delete process.env.OPENCODE_API_KEY + delete process.env.XDG_CONFIG_HOME + }) + + describe("opencode run key rotation (non-interactive subprocess)", () => { + cliIt.live( + "uses key2 when throttle.json marks key1 as throttled at startup", + ({ llm, opencode }) => + Effect.gen(function* () { + // Pre-seed throttle.json: key1 is throttled + const now = Date.now() + await fs.mkdir(path.dirname(throttleFile), { recursive: true }) + await fs.writeFile( + throttleFile, + JSON.stringify([ + { + source: "OPENCODE_API_KEY", + key: "key1", + startTime: now - 1000, + endTime: now + 7_200_000, // 2 hours from now + }, + ]), + ) + + // LLM mock: returns success for any key (key2 will be used) + yield* llm.success("hello from key2") + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1,key2", OPENCODE_THROTTLE_ENABLE: "true" }, + timeoutMs: 30_000, + }) + expect(result.exitCode).toBe(0) + // key2 was used (key1 skipped due to throttle.json pre-check) + }), + 45_000, + ) + + cliIt.live( + "falls back to key2 and writes throttle.json when key1 returns 429", + ({ llm, opencode }) => + Effect.gen(function* () { + // key1: 429, key2: success + yield* llm.errorForKey("key1", 429, { type: "error", error: { type: "RateLimitError", message: "Rate limit" } }) + yield* llm.success("hello from key2") + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1,key2", OPENCODE_THROTTLE_ENABLE: "true" }, + timeoutMs: 40_000, + }) + expect(result.exitCode).toBe(0) + + // throttle.json should now have key1 + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data.some((r: any) => r.key === "key1" && r.source === "OPENCODE_API_KEY")).toBe(true) + }), + 60_000, + ) + + cliIt.live( + "falls back to key2 when key1 returns 401", + ({ llm, opencode }) => + Effect.gen(function* () { + yield* llm.errorForKey("key1", 401, { error: "Unauthorized" }) + yield* llm.success("hello") + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1,key2" }, + timeoutMs: 40_000, + }) + expect(result.exitCode).toBe(0) + + // 401 does NOT write throttle.json + const exists = await fs.stat(throttleFile).then(() => true).catch(() => false) + expect(exists).toBe(false) + }), + 60_000, + ) + + cliIt.live( + "exits nonzero when all keys are throttled", + ({ llm, opencode }) => + Effect.gen(function* () { + const now = Date.now() + await fs.mkdir(path.dirname(throttleFile), { recursive: true }) + await fs.writeFile( + throttleFile, + JSON.stringify([ + { source: "OPENCODE_API_KEY", key: "key1", startTime: now - 1000, endTime: now + 7_200_000 }, + { source: "OPENCODE_API_KEY", key: "key2", startTime: now - 1000, endTime: now + 7_200_000 }, + ]), + ) + + // LLM should not be called at all + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1,key2", OPENCODE_THROTTLE_ENABLE: "true" }, + timeoutMs: 15_000, + }) + expect(result.exitCode).not.toBe(0) + expect(result.durationMs).toBeLessThan(10_000) + }), + 30_000, + ) + + cliIt.live( + "single key still writes throttle.json when rate limited", + ({ llm, opencode }) => + Effect.gen(function* () { + yield* llm.error(429, { type: "error", error: { type: "RateLimitError", message: "Rate limit" } }) + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1", OPENCODE_THROTTLE_ENABLE: "true" }, + timeoutMs: 30_000, + }) + expect(result.exitCode).not.toBe(0) + + // throttle.json written even for single key + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data.some((r: any) => r.key === "key1")).toBe(true) + }), + 45_000, + ) + + cliIt.live( + "throttle disabled: does not read or write throttle.json", + ({ llm, opencode }) => + Effect.gen(function* () { + yield* llm.error(429, { type: "error", error: { type: "RateLimitError", message: "Rate limit" } }) + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1", OPENCODE_THROTTLE_ENABLE: "false" }, + timeoutMs: 30_000, + }) + expect(result.exitCode).not.toBe(0) + const exists = await fs.stat(throttleFile).then(() => true).catch(() => false) + expect(exists).toBe(false) + }), + 45_000, + ) + }) + ``` + +- [ ] **Step 2: Verify the test runner recognizes the new test** + + ```bash + cd packages/opencode + bun test test/cli/run/run-key-rotation.test.ts --list 2>&1 + ``` + + Expected: lists the 6 test cases without errors. + +- [ ] **Step 3: Run E2E tests** + + ```bash + cd packages/opencode + bun test test/cli/run/run-key-rotation.test.ts --timeout 120000 + ``` + + Expected: all 6 tests pass. These are slow (30–60 s each); total runtime ~5 min. + +- [ ] **Step 4: Commit** + + ```bash + cd packages/opencode + git add test/cli/run/run-key-rotation.test.ts + git commit -s -S -m "test(opencode): add E2E tests for key rotation and throttle suppression" + ``` + +--- + +## Spec Coverage Check + +| Spec requirement | Covered by task | +|---|---| +| `OPENCODE_API_KEY=key1,key2` multi-key parsing | Task 3 (parseKeys) | +| throttle.json cross-process persistence | Task 1 (ThrottleStore) | +| Flock-based write locking, crash recovery | Task 1 (uses existing Flock with staleMs) | +| Lockless reads | Task 1 (`isThrottled` — no lock) | +| Throttle on 429/quota error | Task 5 (quota_limit branch) | +| Fallback on 401/invalid key | Task 5 (invalid_key branch) | +| No throttle write on 401 | Task 5 + Task 6 (E2E asserts no throttle file) | +| Single key still writes throttle.json | Task 5 + Task 6 | +| `OPENCODE_THROTTLE_ENABLE=false` disables throttle | Task 3, Task 6 | +| `OPENCODE_THROTTLE_DURATION` configures window | Task 3 | +| `OPENCODE_WELAN_LOG=false` disables welan.txt | Task 2 | +| welan.txt append logging | Task 2, Task 3 | +| Session ID threaded across attempts | Task 5 (overrideSessionID) | +| Server re-initialized between key attempts | Task 5 (Server.Default.reset()) | +| Unit tests for ThrottleStore | Task 1 | +| Unit tests for KeyRotator | Task 3 | +| E2E tests | Task 6 | diff --git a/docs/superpowers/specs/2026-07-09-key-rotation-throttle-design.md b/docs/superpowers/specs/2026-07-09-key-rotation-throttle-design.md new file mode 100644 index 000000000000..e98d1876d861 --- /dev/null +++ b/docs/superpowers/specs/2026-07-09-key-rotation-throttle-design.md @@ -0,0 +1,377 @@ +# Design: OPENCODE_API_KEY Multi-Key Rotation & Throttle Suppression + +**Date**: 2026-07-09 +**Status**: Approved +**Scope**: `packages/opencode` only; OPENCODE_API_KEY provider + +--- + +## Background + +Two prior commits set the foundation: + +- **a617679**: When the API returns a quota/rate-limit error, `opencode run` exits immediately instead of waiting. +- **b35fe2e**: `OPENCODE_API_KEY` env var takes priority over stored keys. + +This feature builds on both to add: +1. **Cross-process throttle suppression** — a shared file records throttled keys so subsequent CLI processes skip them automatically. +2. **In-process multi-key fallback** — `OPENCODE_API_KEY=key1,key2,key3` enables automatic key rotation within the same `opencode run` invocation. +3. **Supplemental rotation log** — key rotation decisions are logged to `~/.config/opencode/welan-log.txt` and to the existing structured opencode log channel. + +--- + +## Architecture Overview + +``` +OPENCODE_API_KEY=key1,key2,key3 + │ + ▼ +┌──────────────────────────────────────────────────────┐ +│ run.ts outer rotation loop (new ~40 lines) │ +│ │ +│ loop: │ +│ key = rotator.selectKey() ◄─────────────┐ │ +│ process.env.OPENCODE_API_KEY = key │ │ +│ runAttempt(args, sessionID) │ │ +│ KeyRotationRetry carries sessionID │ │ +│ │ │ +│ quota_limit → rotator.recordThrottle(key) ────┘ │ +│ invalid_key → rotator.markInvalid(key) ────┘ │ +│ success/other err → break │ +└──────────────────────────────────────────────────────┘ + │ each attempt re-runs full Effect program + ▼ + dispose current directory InstanceState + reset embedded server lazy handler + next local fetch reads updated process.env.OPENCODE_API_KEY + │ + ▼ same SQLite DB on disk + session context preserved across attempts + +┌─────────────────────────────────────┐ +│ KeyRotator (key-rotator.ts) │ +│ - parse comma-separated key list │ +│ - selectKey(): skip throttled + │ +│ in-process invalid keys │ +│ - recordThrottle() / markInvalid() │ +└─────────────────┬───────────────────┘ + │ reads/writes + ▼ +┌─────────────────────────────────────┐ +│ ThrottleStore (throttle-store.ts) │ +│ ~/.config/opencode/throttle.json │ +│ - isThrottled(key): lockless read │ +│ - addThrottle(key): Flock write │ +│ - cleanExpired(key): Flock write │ +└─────────────────────────────────────┘ + + +┌─────────────────────────────────────┐ +│ RotationLogger (rotation-logger.ts)│ +│ ~/.config/opencode/welan-log.txt │ +│ + opencode.log structured format │ +│ OPENCODE_WELAN_LOG=true (default) │ +└─────────────────────────────────────┘ +``` + +### New files + +``` +packages/opencode/src/provider/ + ├── throttle-store.ts (new) + ├── key-rotator.ts (new) + └── rotation-logger.ts (new) + +packages/opencode/test/provider/ + ├── throttle-store.test.ts (new) + └── key-rotator.test.ts (new) + +packages/opencode/test/cli/run/ + └── run-key-rotation.test.ts (new) +``` + +### Modified files + +``` +packages/opencode/src/cli/cmd/run.ts (~40 lines: outer rotation loop) +packages/opencode/src/session/retry.ts (~5 lines: add isInvalidKeyAPIError) +``` + +--- + +## Environment Variables + +| Variable | Default | Description | +|---|---|---| +| `OPENCODE_API_KEY` | — | Comma-separated key list: `key1,key2,key3`. Single key = existing behavior. | +| `OPENCODE_THROTTLE_ENABLE` | `"true"` | Set to `"false"` to disable throttle check entirely. | +| `OPENCODE_THROTTLE_DURATION` | `"120"` | Throttle window in minutes. | +| `OPENCODE_WELAN_LOG` | `"true"` | Set to `"false"` to disable welan.txt output. | + +--- + +## Component: ThrottleStore + +**File**: `packages/opencode/src/provider/throttle-store.ts` +**Dependencies**: Node.js `fs/promises`, existing `Flock` utility, `Global.Path.config` + +### throttle.json format + +```json +[ + { + "source": "OPENCODE_API_KEY", + "key": "sk-abc123...", + "startTime": 1720513264000, + "endTime": 1720520464000 + } +] +``` + +`source` holds the env var name, enabling future extension to other providers without schema changes. + +### API + +```typescript +function isThrottled(source: string, key: string): Promise +// Lockless read. +// → true if record exists AND now < endTime +// → false if record not found +// → false if record exists but now >= endTime (triggers async cleanExpired, non-blocking) +// → false on any read error (prefer false positive to blocking API calls) + +function addThrottle(source: string, key: string, durationMinutes: number): Promise +// Flock → re-read → upsert → write → release. +// On timeout (2s): silently return. Never blocks caller beyond 2s. + +function cleanExpired(source: string, key: string): Promise +// Flock → re-read → remove expired entries for key → write → release. +``` + +### Locking strategy + +Uses existing `Flock.acquire("throttle-store", { dir: Global.Path.config, timeoutMs: 2_000, staleMs: 10_000 })`. + +- **Reads**: no lock — a stale read at worst causes one extra API call that also rate-limits, which then updates the file. +- **Writes**: Flock handles crash recovery via heartbeat + stale detection (no orphaned locks on process crash). +- **Timeout**: 2 s; on timeout `addThrottle` returns without writing. Principle: *prefer more API calls over fewer*. + +--- + +## Component: KeyRotator + +**File**: `packages/opencode/src/provider/key-rotator.ts` +**Dependencies**: `ThrottleStore`, `RotationLogger`, env vars only — no Effect, no Provider internals. + +### API + +```typescript +function parseKeys(): string[] +// OPENCODE_API_KEY="key1,key2,key3" → ["key1", "key2", "key3"] +// Trims, de-duplicates, drops empty strings. + +class KeyRotator { + constructor(keys: string[]) + + // Returns first key not in invalid set and not throttled. + // Returns null if all keys are exhausted. + async selectKey(): Promise + + // Writes throttle record for key. Async, non-blocking. + async recordThrottle(key: string): Promise + + // Marks key as invalid for this process lifetime. Does not write throttle.json. + markInvalid(key: string): void + + // Returns true if any key other than currentKey is not invalid. + hasAlternative(currentKey: string): boolean +} +``` + +### `selectKey()` algorithm + +``` +for key in this.keys: + if key in this.invalid → skip + if OPENCODE_THROTTLE_ENABLE !== "false": + if await ThrottleStore.isThrottled("OPENCODE_API_KEY", key) → log, skip + return key +return null +``` + +### Key masking in logs + +Last 6 characters only: `***abc123`. Balances debuggability with security. + +--- + +## Component: RotationLogger + +**File**: `packages/opencode/src/provider/rotation-logger.ts` +**Dependencies**: Node.js `fs/promises`, `Global.Path.config` — no Effect. + +```typescript +type Level = "info" | "warn" | "error" + +function log(level: Level, message: string): void +// Fire-and-forget fs.appendFile to ~/.config/opencode/welan-log.txt. +// Also appends the same event to opencode.log using the existing structured log format. +// No-op if OPENCODE_WELAN_LOG === "false". +// Errors silently swallowed (never block API calls). +``` + +Example log lines: +``` +[2026-07-09T18:41:04+08:00] [INFO] key-rotation: start, 3 keys configured +[2026-07-09T18:41:04+08:00] [INFO] key-rotation: attempt 1/3 with key ***abc123 +[2026-07-09T18:41:09+08:00] [WARN] key-rotation: key ***abc123 quota_limit, throttle until 2026-07-09T20:41:09+08:00 +[2026-07-09T18:41:09+08:00] [INFO] key-rotation: attempt 2/3 with key ***xyz456 +[2026-07-09T18:41:15+08:00] [INFO] key-rotation: succeeded with key ***xyz456 +``` + +**Relationship with existing Effect Logger**: Inside the Effect program, existing `Effect.logInfo/logWarn` calls write to `opencode.log` as before. RotationLogger is called from outside the Effect boundary, so it appends matching structured lines to `opencode.log` directly and mirrors to stderr when `OPENCODE_PRINT_LOGS=1`. + +--- + +## Component: run.ts integration + +### New function in retry.ts + +```typescript +// ~5 lines, mirrors isQuotaOrRateLimitAPIError +export function isInvalidKeyAPIError(error: unknown): boolean +// → true if error.name === "APIError" && error.data.statusCode === 401 +``` + +### KeyRotationRetry signal + +`execute(sdk)` keeps its existing return shape. Only key-rotation-worthy errors throw a private signal: + +```typescript +class KeyRotationRetry extends Error { + constructor( + readonly reason: "quota_limit" | "invalid_key", + readonly sessionID: string, + readonly detail?: string, + ) +} +``` + +| Trigger | `reason` | Write throttle? | Try next key? | +|---|---|---|---| +| `isQuotaOrRateLimitAPIError` | `quota_limit` | ✅ | ✅ | +| `isQuotaOrRateLimitRetryStatus` | `quota_limit` | ✅ | ✅ | +| `isInvalidKeyAPIError` (401) | `invalid_key` | ❌ | ✅ | +| session idle | none | — | — | +| other error | none | — | — | + +### Outer rotation loop (new `runWithKeyRotation`) + +```typescript +async function runWithKeyRotation(args: RunArgs): Promise { + const keys = KeyRotator.parseKeys() + const rotator = new KeyRotator(keys) + let sessionID: string | undefined = args.session + RotationLogger.log("info", `start, ${keys.length} key(s) configured`) + + while (true) { + const key = await rotator.selectKey() + if (!key) { + RotationLogger.log("error", "all OPENCODE_API_KEY keys exhausted or throttled") + break + } + + process.env.OPENCODE_API_KEY = key + RotationLogger.log("info", `attempt with key ***${key.slice(-6)}`) + + try { + await runAttempt({ ...args, session: sessionID }) + return undefined + } catch (error) { + if (!(error instanceof KeyRotationRetry)) throw error + sessionID = error.sessionID + if (error.reason === "quota_limit") { + await rotator.recordThrottle(key) + if (!rotator.hasAlternative(key)) return error.detail + RotationLogger.log("warn", `key ***${key.slice(-6)} throttled, trying next`) + continue + } + + rotator.markInvalid(key) + if (!rotator.hasAlternative(key)) return error.detail + RotationLogger.log("warn", `key ***${key.slice(-6)} invalid, trying next`) + continue + } + } + return undefined +} +``` + +The `effectCmd` handler becomes: + +```typescript +handler: Effect.fn("Cli.run")(function*(args) { + // ... existing layer setup (unchanged) ... + yield* Effect.promise(() => runWithKeyRotation(parsedArgs)) +}) +``` + +### Per-attempt local cache reset + +Each retry keeps the existing embedded-server process but resets only the local state needed for a new key: + +- **InstanceState for the current directory is disposed** so Provider/Env state is rebuilt. +- **Embedded server lazy handler is reset** so subsequent local fetches use the refreshed state. +- **SQLite DB persists** on disk — session messages and history survive. +- **Session ID threading** — first attempt creates session S; subsequent attempts pass S explicitly, triggering V2 exact-retry semantics so history is available. + +--- + +## Behavior Summary + +| Scenario | Behavior | +|---|---| +| Key in throttle.json, window active | `selectKey()` skips; no API call made | +| Key in throttle.json, window expired | Record cleaned; key used normally | +| Key returns 429 at runtime | Throttle recorded; next key tried | +| Key returns 401 at runtime | Key marked invalid in-process only; next key tried | +| All keys exhausted | Last error returned; exit nonzero | +| Single key configured | Throttle pre-check and throttle write still apply; no key rotation (no alternatives) | +| `OPENCODE_THROTTLE_ENABLE=false` | Throttle file reads/writes skipped; multi-key invalid-key fallback still active | + +--- + +## Testing Strategy + +### Unit tests + +**`throttle-store.test.ts`** — pure async, temp dir, no Effect: +- `isThrottled`: absent → false; active → true; expired → false + async cleanup +- `addThrottle`: creates file; upserts existing; handles concurrent access via Flock +- Stale lock: simulate stale (delete heartbeat); verify next caller acquires successfully + +**`key-rotator.test.ts`** — pure async, mock `ThrottleStore.isThrottled`: +- `parseKeys`: single key, multi-key, empty, whitespace trimming, dedup +- `selectKey`: first available; skips invalid; skips throttled; null when all exhausted +- `recordThrottle`: calls `addThrottle` with correct `source` and `durationMinutes` +- `markInvalid`: skips key in subsequent calls; does not write throttle.json +- `hasAlternative`: true when non-invalid alternatives exist; false when all invalid + +### End-to-end tests + +**`run-key-rotation.test.ts`** — extends `cliIt.live` pattern: +- **Throttle pre-check**: seed `throttle.json` with active record for key1; verify key2 used. +- **Quota fallback**: key1 returns 429; verify key2 tried; verify `throttle.json` updated. +- **Invalid key fallback**: key1 returns 401; verify key2 tried; no throttle record written. +- **All keys exhausted**: both keys throttled; verify exit nonzero; zero API calls made. +- **Single key fast path**: verify no overhead when `OPENCODE_API_KEY` has one value. +- **Session continuity**: key1 429 creates session S; key2 reuses session S; history present. + +--- + +## Cherry-Pick Constraints + +- `throttle-store.ts`, `key-rotator.ts`, `rotation-logger.ts` — **new files, zero imports from Provider/Session/Effect layers**. +- `run.ts` — outer loop is a **new function** `runWithKeyRotation`; existing `execute()` keeps its return shape and only throws `KeyRotationRetry` for rotation-worthy errors. +- `retry.ts` — adds one function `isInvalidKeyAPIError`; no existing functions modified. +- rotation log appends are fire-and-forget; failures are silently swallowed. +- `OPENCODE_THROTTLE_ENABLE=false`: throttle file reads/writes are bypassed; the rotation loop itself still runs for multi-key invalid-key fallback. diff --git a/packages/opencode/src/cli/cmd/run.ts b/packages/opencode/src/cli/cmd/run.ts index a5f64991af19..13fe06a80542 100644 --- a/packages/opencode/src/cli/cmd/run.ts +++ b/packages/opencode/src/cli/cmd/run.ts @@ -26,9 +26,22 @@ import { createOpencodeClient, type OpencodeClient, type ToolPart } from "@openc import { FormatError, FormatUnknownError } from "../error" import { INTERACTIVE_INPUT_ERROR, resolveInteractiveStdin } from "./run/runtime.stdin" import { SessionRetry } from "@/session/retry" +import { KeyRotator, parseKeys } from "@/provider/key-rotator" +import { KeyRotationRetry } from "@/provider/key-rotation-retry" +import { RotationLogger } from "@/provider/rotation-logger" +import { disposeInstance } from "@/effect/instance-registry" type ModelInput = Parameters[0]["model"] +function keyRotationActive() { + return process.env.OPENCODE_KEY_ROTATION_ACTIVE === "true" +} + +function throwKeyRotation(error: unknown) { + if (SessionRetry.isQuotaOrRateLimitAPIError(error)) throw new KeyRotationRetry("quota_limit") + if (SessionRetry.isInvalidKeyAPIError(error)) throw new KeyRotationRetry("invalid_key") +} + function pick(value: string | undefined): ModelInput | undefined { if (!value) return undefined const [providerID, ...rest] = value.split("/") @@ -784,10 +797,12 @@ export const RunCommand = effectCmd({ error = error ? error + EOL + err : err const limit = SessionRetry.isQuotaOrRateLimitAPIError(props.error) if (emit("error", { error: props.error })) { + if (keyRotationActive()) throwKeyRotation(props.error) if (limit) return error continue } UI.error(err) + if (keyRotationActive()) throwKeyRotation(props.error) if (limit) return error } @@ -795,6 +810,10 @@ export const RunCommand = effectCmd({ const status = event.properties.status if (status.type === "retry" && SessionRetry.isQuotaOrRateLimitRetryStatus(status)) { error = error ? error + EOL + status.message : status.message + if (keyRotationActive()) { + if (!emit("error", { error: status })) UI.error(status.message) + throw new KeyRotationRetry("quota_limit") + } if (emit("error", { error: status })) return error UI.error(status.message) return error @@ -839,6 +858,7 @@ export const RunCommand = effectCmd({ if (!interactive) { const events = await client.event.subscribe() const completed = loop(client, events).catch((e) => { + if (e instanceof KeyRotationRetry) throw e console.error(e) process.exitCode = 1 }) @@ -859,6 +879,7 @@ export const RunCommand = effectCmd({ }) if (result.error) { if (!emit("error", { error: result.error })) UI.error(formatRunError(result.error)) + if (keyRotationActive()) throwKeyRotation(result.error) process.exitCode = 1 return } @@ -876,6 +897,7 @@ export const RunCommand = effectCmd({ }) if (result.error) { if (!emit("error", { error: result.error })) UI.error(formatRunError(result.error)) + if (keyRotationActive()) throwKeyRotation(result.error) process.exitCode = 1 return } @@ -907,7 +929,76 @@ export const RunCommand = effectCmd({ } catch (error) { dieInteractive(error) } - return + } + + async function runWithKeyRotation(createSdk: () => OpencodeClient): Promise { + const { Server } = await import("@/server/server") + const keys = parseKeys() + if (keys.length <= 1) { + await execute(createSdk()) + return + } + const rotator = new KeyRotator(keys) + + RotationLogger.log("info", `key-rotation: start, ${keys.length} key(s) configured`) + + let attempt = 0 + while (true) { + const key = await rotator.selectKey() + if (!key) { + RotationLogger.log("error", "key-rotation: all OPENCODE_API_KEY keys exhausted or throttled") + process.exitCode = 1 + return + } + + attempt++ + process.env.OPENCODE_API_KEY = key + if (attempt > 1) { + await disposeInstance(directory ?? root) + Server.Default.reset() + } + RotationLogger.log("info", `key-rotation: attempt ${attempt} with key ***${key.slice(-6)}`) + + const previousRotationActive = process.env.OPENCODE_KEY_ROTATION_ACTIVE + process.env.OPENCODE_KEY_ROTATION_ACTIVE = "true" + try { + try { + await execute(createSdk()) + } catch (error) { + if (!(error instanceof KeyRotationRetry)) throw error + if (error.reason === "quota_limit") { + await rotator.recordThrottle(key) + RotationLogger.log( + "warn", + `key-rotation: key ***${key.slice(-6)} quota_limit, ${rotator.hasAlternative(key) ? "trying next" : "no more keys"}`, + ) + if (!rotator.hasAlternative(key)) { + process.exitCode = 1 + return + } + process.exitCode = 0 + continue + } + rotator.markInvalid(key) + RotationLogger.log( + "warn", + `key-rotation: key ***${key.slice(-6)} invalid, ${rotator.hasAlternative(key) ? "trying next" : "no more keys"}`, + ) + if (!rotator.hasAlternative(key)) { + process.exitCode = 1 + return + } + process.exitCode = 0 + continue + } + if (process.exitCode) return + RotationLogger.log("info", `key-rotation: success with key ***${key.slice(-6)}`) + return + } finally { + if (previousRotationActive === undefined) delete process.env.OPENCODE_KEY_ROTATION_ACTIVE + else process.env.OPENCODE_KEY_ROTATION_ACTIVE = previousRotationActive + } + } } if (interactive && !args.attach && !args.session && !args.continue) { @@ -959,12 +1050,10 @@ export const RunCommand = effectCmd({ if (auth) headers.set("Authorization", auth) return Server.Default().app.fetch(new Request(request, { headers })) }) as typeof globalThis.fetch - const sdk = createOpencodeClient({ - baseUrl: "http://opencode.internal", - fetch: fetchFn, - directory, - }) - await execute(sdk) + + const createSdk = () => createOpencodeClient({ baseUrl: "http://opencode.internal", fetch: fetchFn, directory }) + + await runWithKeyRotation(createSdk) }) }), }) diff --git a/packages/opencode/src/provider/key-rotation-retry.ts b/packages/opencode/src/provider/key-rotation-retry.ts new file mode 100644 index 000000000000..f5b18a3de90d --- /dev/null +++ b/packages/opencode/src/provider/key-rotation-retry.ts @@ -0,0 +1,9 @@ +export type RotateReason = "quota_limit" | "invalid_key" + +export class KeyRotationRetry extends Error { + override readonly name = "KeyRotationRetry" + + constructor(readonly reason: RotateReason) { + super(`key rotation: ${reason}`) + } +} diff --git a/packages/opencode/src/provider/key-rotator.ts b/packages/opencode/src/provider/key-rotator.ts new file mode 100644 index 000000000000..34370f55932c --- /dev/null +++ b/packages/opencode/src/provider/key-rotator.ts @@ -0,0 +1,73 @@ +import { ThrottleStore } from "./throttle-store" +import { RotationLogger } from "./rotation-logger" + +const SOURCE = "OPENCODE_API_KEY" +type KeyRotatorStore = Pick + +export function parseKeys(): string[] { + const raw = process.env.OPENCODE_API_KEY ?? "" + return raw + .split(",") + .map((k) => k.trim()) + .filter(Boolean) +} + +function throttleDuration(): number { + const raw = process.env.OPENCODE_THROTTLE_DURATION + const parsed = raw ? parseInt(raw, 10) : NaN + return Number.isFinite(parsed) && parsed > 0 ? parsed : 120 +} + +function isThrottleEnabled(): boolean { + return process.env.OPENCODE_THROTTLE_ENABLE !== "false" +} + +function maskKey(key: string): string { + return `***${key.slice(-6)}` +} + +export class KeyRotator { + private readonly invalid = new Set() + + constructor( + private readonly keys: string[], + private readonly store: KeyRotatorStore = ThrottleStore, + ) {} + + async selectKey(): Promise { + for (const key of this.keys) { + if (this.invalid.has(key)) continue + if (isThrottleEnabled() && (await this.store.isThrottled(SOURCE, key))) { + RotationLogger.log("info", `key-rotation: key ${maskKey(key)} is throttled, skipping`) + continue + } + return key + } + return null + } + + async recordThrottle(key: string): Promise { + if (!isThrottleEnabled()) return + const duration = throttleDuration() + RotationLogger.log( + "warn", + `key-rotation: key ${maskKey(key)} throttled for ${duration} minutes, writing throttle record`, + ) + await this.store.addThrottle(SOURCE, key, duration) + } + + markInvalid(key: string): void { + RotationLogger.log("warn", `key-rotation: key ${maskKey(key)} marked invalid, skipping for this process`) + this.invalid.add(key) + } + + hasAlternative(currentKey: string): boolean { + return this.keys.some((k) => k !== currentKey && !this.invalid.has(k)) + } +} + +const _parseKeys = parseKeys + +export namespace KeyRotator { + export const parseKeys = _parseKeys +} diff --git a/packages/opencode/src/provider/rotation-logger.ts b/packages/opencode/src/provider/rotation-logger.ts new file mode 100644 index 000000000000..2376d6727d25 --- /dev/null +++ b/packages/opencode/src/provider/rotation-logger.ts @@ -0,0 +1,64 @@ +import path from "path" +import fs from "fs/promises" +import { Global } from "@opencode-ai/core/global" +import { runID } from "@opencode-ai/core/observability/shared" + +export type Level = "info" | "warn" | "error" + +const PID_PREFIX = new Date().toISOString().slice(11, 23).replace(/[:.]/g, "") // e.g. "012300000Z" + +type LoggerOptions = { + configDir: string | (() => string) + logDir: string | (() => string) +} + +export function createRotationLogger(options: LoggerOptions) { + const configDir = () => (typeof options.configDir === "string" ? options.configDir : options.configDir()) + const logDir = () => (typeof options.logDir === "string" ? options.logDir : options.logDir()) + const filePath = () => path.join(configDir(), "welan-log.txt") + const regularLogPath = () => path.join(logDir(), "opencode.log") + let writeChain = Promise.resolve() + + function log(level: Level, message: string): void { + if (process.env.OPENCODE_WELAN_LOG === "false") return + const now = new Date() + const timestamp = + now.toLocaleString("sv-SE", { hour12: false }).replace(" ", "T") + + "." + + String(now.getMilliseconds()).padStart(3, "0") + const line = `[${PID_PREFIX}] [${timestamp}] [${level.toUpperCase()}] ${message}\n` + const regularLine = regularLogLine( + level, + message.startsWith("key-rotation:") ? message : `key-rotation: ${message}`, + now, + ) + writeChain = writeChain + .then(() => fs.mkdir(path.dirname(filePath()), { recursive: true })) + .then(() => fs.appendFile(filePath(), line)) + .then(() => fs.mkdir(path.dirname(regularLogPath()), { recursive: true })) + .then(() => fs.appendFile(regularLogPath(), regularLine)) + .then(() => { + if (process.env.OPENCODE_PRINT_LOGS === "1") process.stderr.write(regularLine) + }) + .catch(() => {}) + } + + return { log } +} + +function regularLogLine(level: Level, message: string, timestamp: Date) { + return `timestamp=${timestamp.toISOString()} level=${level.toUpperCase()} run=${runID} message=${formatLogValue(message)}\n` +} + +function formatLogValue(value: string) { + return /^[^\s="\\]+$/.test(value) ? value : JSON.stringify(value) +} + +const logger = createRotationLogger({ + configDir: () => Global.Path.config, + logDir: () => Global.Path.log, +}) + +export const log = logger.log + +export * as RotationLogger from "./rotation-logger" diff --git a/packages/opencode/src/provider/throttle-store.ts b/packages/opencode/src/provider/throttle-store.ts new file mode 100644 index 000000000000..2eb8b573c9a9 --- /dev/null +++ b/packages/opencode/src/provider/throttle-store.ts @@ -0,0 +1,102 @@ +import path from "path" +import fs from "fs/promises" +import { Flock } from "@opencode-ai/core/util/flock" +import { Global } from "@opencode-ai/core/global" + +type ThrottleRecord = { + source: string + key: string + startTime: number + endTime: number +} + +type StoreOptions = { + configDir: string | (() => string) +} + +export function createThrottleStore(options: StoreOptions) { + const configDir = () => (typeof options.configDir === "string" ? options.configDir : options.configDir()) + const filePath = () => path.join(configDir(), "throttle.json") + + async function readRecords(): Promise { + try { + const raw = await fs.readFile(filePath(), "utf8") + const parsed = JSON.parse(raw) + if (!Array.isArray(parsed)) return [] + return parsed as ThrottleRecord[] + } catch { + return [] + } + } + + async function writeRecords(records: ThrottleRecord[]): Promise { + await fs.mkdir(path.dirname(filePath()), { recursive: true }) + await fs.writeFile(filePath(), JSON.stringify(records, null, 2)) + } + + async function isThrottled(source: string, key: string): Promise { + const records = await readRecords() + const now = Date.now() + const record = records.find((r) => r.source === source && r.key === key) + if (!record) return false + if (now < record.endTime) return true + void cleanExpired(source, key) + return false + } + + async function addThrottle(source: string, key: string, durationMinutes: number): Promise { + let lease: Awaited> | null = null + try { + lease = await Flock.acquire("throttle-store", { + dir: configDir(), + timeoutMs: 2_000, + staleMs: 10_000, + }) + } catch { + return + } + try { + const records = await readRecords() + const now = Date.now() + const endTime = now + durationMinutes * 60 * 1000 + const idx = records.findIndex((r) => r.source === source && r.key === key) + const record: ThrottleRecord = { source, key, startTime: now, endTime } + if (idx >= 0) records[idx] = record + else records.push(record) + await writeRecords(records) + } finally { + await lease.release() + } + } + + async function cleanExpired(source: string, key: string): Promise { + let lease: Awaited> | null = null + try { + lease = await Flock.acquire("throttle-store", { + dir: configDir(), + timeoutMs: 2_000, + staleMs: 10_000, + }) + } catch { + return + } + try { + const records = await readRecords() + const now = Date.now() + const filtered = records.filter((r) => !(r.source === source && r.key === key && now >= r.endTime)) + await writeRecords(filtered) + } finally { + await lease.release() + } + } + + return { isThrottled, addThrottle, cleanExpired } +} + +const store = createThrottleStore({ configDir: () => Global.Path.config }) + +export const isThrottled = store.isThrottled +export const addThrottle = store.addThrottle +export const cleanExpired = store.cleanExpired + +export * as ThrottleStore from "./throttle-store" diff --git a/packages/opencode/src/session/retry.ts b/packages/opencode/src/session/retry.ts index f89b3bfd257c..73980e0ae984 100644 --- a/packages/opencode/src/session/retry.ts +++ b/packages/opencode/src/session/retry.ts @@ -53,14 +53,28 @@ export function isQuotaOrRateLimitPayload(value: unknown): boolean { return false } +type SerializedSessionAPIError = { + name: "APIError" + data: Record +} + +function isSerializedSessionAPIError(error: unknown): error is SerializedSessionAPIError { + return isRecord(error) && error.name === "APIError" && isRecord(error.data) +} + export function isQuotaOrRateLimitAPIError(error: unknown): boolean { - if (!isRecord(error) || error.name !== "APIError" || !isRecord(error.data)) return false + if (!isSerializedSessionAPIError(error)) return false if (error.data.statusCode !== 429) return false const body = text(error.data.responseBody) if (isQuotaOrRateLimitPayload(parseJSON(body))) return true return /insufficient[-_\s]?quota|quota[-_\s]?exceeded/i.test(body) } +export function isInvalidKeyAPIError(error: unknown): boolean { + if (!isSerializedSessionAPIError(error)) return false + return error.data.statusCode === 401 +} + export function isQuotaOrRateLimitRetryStatus(status: unknown): boolean { if (!isRecord(status) || status.type !== "retry") return false if (isRecord(status.action)) { @@ -110,6 +124,32 @@ export function delay(attempt: number, error?: SessionV1.APIError) { export function retryable(error: Err, provider: string) { // context overflow errors should not be retried if (SessionV1.ContextOverflowError.isInstance(error)) return undefined + + if (process.env.OPENCODE_KEY_ROTATION_ACTIVE === "true" && process.env.OPENCODE_THROTTLE_ENABLE !== "false") { + const isQuota = + isQuotaOrRateLimitAPIError(error) || + (() => { + const msg = isRecord(error.data) ? error.data.message : undefined + if (typeof msg === "string") { + const lower = msg.toLowerCase() + return ( + lower.includes("rate increased too quickly") || + lower.includes("rate limit") || + lower.includes("too many requests") + ) + } + const json = parseJSON(msg) + if (json && typeof json === "object") { + const code = typeof json.code === "string" ? json.code : "" + if (json.type === "error" && json.error?.type === "too_many_requests") return true + if (code.includes("exhausted") || code.includes("unavailable")) return true + if (json.type === "error" && typeof json.error?.code === "string" && json.error.code.includes("rate_limit")) + return true + } + return false + })() + if (isQuota) return undefined + } if (SessionV1.APIError.isInstance(error)) { const status = error.data.statusCode // 5xx errors are transient server failures and should always be retried, diff --git a/packages/opencode/test/cli/run/run-key-rotation.test.ts b/packages/opencode/test/cli/run/run-key-rotation.test.ts new file mode 100644 index 000000000000..4f6be52509b8 --- /dev/null +++ b/packages/opencode/test/cli/run/run-key-rotation.test.ts @@ -0,0 +1,175 @@ +// Key rotation behavior for `opencode run` (non-interactive subprocess). +// Kept in a separate file to avoid competing with the main CLI regression suite. +import { afterEach, beforeEach, describe, expect } from "bun:test" +import path from "path" +import fs from "fs/promises" +import { Effect } from "effect" +import { cliIt } from "../../lib/cli-process" + +beforeEach(() => { + delete process.env.OPENCODE_API_KEY +}) + +afterEach(() => { + delete process.env.OPENCODE_API_KEY +}) + +describe("opencode run key rotation (non-interactive subprocess)", () => { + cliIt.live( + "uses key2 when throttle.json marks key1 as throttled at startup", + ({ llm, opencode, home }) => + Effect.gen(function* () { + const throttleFile = path.join(home, ".config", "opencode", "throttle.json") + // Pre-seed throttle.json: key1 is throttled + const now = Date.now() + yield* Effect.promise(() => fs.mkdir(path.dirname(throttleFile), { recursive: true })) + yield* Effect.promise(() => + fs.writeFile( + throttleFile, + JSON.stringify([ + { + source: "OPENCODE_API_KEY", + key: "key1", + startTime: now - 1000, + endTime: now + 7_200_000, // 2 hours from now + }, + ]), + ), + ) + + // LLM mock: returns success for any key (key2 will be used) + yield* llm.success("hello from key2") + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1,key2", OPENCODE_THROTTLE_ENABLE: "true" }, + timeoutMs: 30_000, + }) + expect(result.exitCode).toBe(0) + // key2 was used (key1 skipped due to throttle.json pre-check) + }), + 45_000, + ) + + cliIt.live( + "falls back to key2 and writes throttle.json when key1 returns 429", + ({ llm, opencode, home }) => + Effect.gen(function* () { + const throttleFile = path.join(home, ".config", "opencode", "throttle.json") + // key1: 429, key2: success + yield* llm.errorForKey("key1", 429, { type: "error", error: { type: "RateLimitError", message: "Rate limit" } }) + yield* llm.success("hello from key2") + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1,key2", OPENCODE_THROTTLE_ENABLE: "true" }, + timeoutMs: 40_000, + }) + expect(result.exitCode).toBe(0) + + // throttle.json should now have key1 + const data = JSON.parse(yield* Effect.promise(() => fs.readFile(throttleFile, "utf8"))) + expect(data.some((r: any) => r.key === "key1" && r.source === "OPENCODE_API_KEY")).toBe(true) + }), + 60_000, + ) + + cliIt.live( + "falls back to key2 when key1 returns 401", + ({ llm, opencode, home }) => + Effect.gen(function* () { + const throttleFile = path.join(home, ".config", "opencode", "throttle.json") + yield* llm.errorForKey("key1", 401, { error: "Unauthorized" }) + yield* llm.success("hello") + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1,key2" }, + timeoutMs: 40_000, + }) + expect(result.exitCode).toBe(0) + + // 401 does NOT write throttle.json + const exists = yield* Effect.promise(() => + fs + .stat(throttleFile) + .then(() => true) + .catch(() => false), + ) + expect(exists).toBe(false) + }), + 60_000, + ) + + cliIt.live( + "exits nonzero when all keys are throttled", + ({ llm, opencode, home }) => + Effect.gen(function* () { + const throttleFile = path.join(home, ".config", "opencode", "throttle.json") + const now = Date.now() + yield* Effect.promise(() => fs.mkdir(path.dirname(throttleFile), { recursive: true })) + yield* Effect.promise(() => + fs.writeFile( + throttleFile, + JSON.stringify([ + { source: "OPENCODE_API_KEY", key: "key1", startTime: now - 1000, endTime: now + 7_200_000 }, + { source: "OPENCODE_API_KEY", key: "key2", startTime: now - 1000, endTime: now + 7_200_000 }, + ]), + ), + ) + + // LLM should not be called at all + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1,key2", OPENCODE_THROTTLE_ENABLE: "true" }, + timeoutMs: 15_000, + }) + expect(result.exitCode).not.toBe(0) + expect(result.durationMs).toBeLessThan(10_000) + }), + 30_000, + ) + + cliIt.live( + "single key exits nonzero without writing throttle.json when rate limited", + ({ llm, opencode, home }) => + Effect.gen(function* () { + const throttleFile = path.join(home, ".config", "opencode", "throttle.json") + yield* llm.error(429, { type: "error", error: { type: "RateLimitError", message: "Rate limit" } }) + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1", OPENCODE_THROTTLE_ENABLE: "true" }, + timeoutMs: 30_000, + }) + expect(result.exitCode).not.toBe(0) + + const exists = yield* Effect.promise(() => + fs + .stat(throttleFile) + .then(() => true) + .catch(() => false), + ) + expect(exists).toBe(false) + }), + 45_000, + ) + + cliIt.live( + "throttle disabled: does not read or write throttle.json", + ({ llm, opencode, home }) => + Effect.gen(function* () { + const throttleFile = path.join(home, ".config", "opencode", "throttle.json") + yield* llm.error(429, { type: "error", error: { type: "RateLimitError", message: "Rate limit" } }) + + const result = yield* opencode.run("say hi", { + env: { OPENCODE_API_KEY: "key1", OPENCODE_THROTTLE_ENABLE: "false" }, + timeoutMs: 30_000, + }) + expect(result.exitCode).not.toBe(0) + const exists = yield* Effect.promise(() => + fs + .stat(throttleFile) + .then(() => true) + .catch(() => false), + ) + expect(exists).toBe(false) + }), + 45_000, + ) +}) diff --git a/packages/opencode/test/lib/cli-process.ts b/packages/opencode/test/lib/cli-process.ts index 12e8d9c866a5..7b3b5325aa75 100644 --- a/packages/opencode/test/lib/cli-process.ts +++ b/packages/opencode/test/lib/cli-process.ts @@ -74,6 +74,7 @@ function isolatedEnv(home: string, configJson: string): Record { OPENCODE_DISABLE_AUTOCOMPACT: "1", OPENCODE_DISABLE_MODELS_FETCH: "1", OPENCODE_AUTH_CONTENT: "{}", + OPENCODE_API_KEY: "test-key", } } diff --git a/packages/opencode/test/lib/llm-server.ts b/packages/opencode/test/lib/llm-server.ts index 245acc7280f5..7d1ba73581b3 100644 --- a/packages/opencode/test/lib/llm-server.ts +++ b/packages/opencode/test/lib/llm-server.ts @@ -18,6 +18,7 @@ type Flow = type Hit = { url: URL body: Record + headers: Record } type Match = (hit: Hit) => boolean @@ -597,10 +598,11 @@ function item(input: Item | Reply) { return input instanceof Reply ? input.item() : input } -function hit(url: string, body: unknown) { +function hit(url: string, body: unknown, headers?: Record) { return { url: new URL(url, "http://localhost"), body: body && typeof body === "object" ? (body as Record) : {}, + headers: headers ?? {}, } satisfies Hit } @@ -622,6 +624,8 @@ namespace TestLLMServer { readonly reason: (value: string, opts?: { text?: string; usage?: Usage }) => Effect.Effect readonly fail: (message?: unknown) => Effect.Effect readonly error: (status: number, body: unknown) => Effect.Effect + readonly success: (value: string, opts?: { usage?: Usage }) => Effect.Effect + readonly errorForKey: (key: string, status: number, body: unknown) => Effect.Effect readonly hang: Effect.Effect readonly hold: (value: string, wait: PromiseLike) => Effect.Effect readonly reset: Effect.Effect @@ -665,14 +669,16 @@ export class TestLLMServer extends Context.Service !entry.match || entry.match(hit)) if (index === -1) return const first = list[index] - list = [...list.slice(0, index), ...list.slice(index + 1)] + if (!first.match) { + list = [...list.slice(0, index), ...list.slice(index + 1)] + } return first.item } const handle = Effect.fn("TestLLMServer.handle")(function* (mode: "chat" | "responses") { const req = yield* HttpServerRequest.HttpServerRequest const body = yield* req.json.pipe(Effect.orElseSucceed(() => ({}))) - const current = hit(req.originalUrl, body) + const current = hit(req.originalUrl, body, req.headers) if (isTitleRequest(body)) { hits = [...hits, current] yield* notify() @@ -750,6 +756,18 @@ export class TestLLMServer extends Context.Service { + const auth = h.headers["authorization"] ?? "" + return auth === `Bearer ${key}` || auth === key + } + queueMatch(match, httpError(status, body)) + }), hang: Effect.gen(function* () { queue(reply().hang().item()) }).pipe(Effect.withSpan("TestLLMServer.hang")), diff --git a/packages/opencode/test/lib/test-provider.ts b/packages/opencode/test/lib/test-provider.ts index cfb5a93e33e5..dd882ba0901e 100644 --- a/packages/opencode/test/lib/test-provider.ts +++ b/packages/opencode/test/lib/test-provider.ts @@ -14,7 +14,7 @@ export function testProviderConfig(llmUrl: string) { test: { name: "Test", id: "test", - env: [], + env: ["OPENCODE_API_KEY"], npm: "@ai-sdk/openai-compatible", models: { "test-model": { @@ -30,7 +30,7 @@ export function testProviderConfig(llmUrl: string) { options: {}, }, }, - options: { apiKey: "test-key", baseURL: llmUrl }, + options: { baseURL: llmUrl }, }, }, } diff --git a/packages/opencode/test/provider/key-rotation-retry.test.ts b/packages/opencode/test/provider/key-rotation-retry.test.ts new file mode 100644 index 000000000000..af82c61a7285 --- /dev/null +++ b/packages/opencode/test/provider/key-rotation-retry.test.ts @@ -0,0 +1,12 @@ +import { describe, expect, it } from "bun:test" +import { KeyRotationRetry } from "../../src/provider/key-rotation-retry" + +describe("KeyRotationRetry", () => { + it("carries reason", () => { + const error = new KeyRotationRetry("quota_limit") + expect(error).toBeInstanceOf(KeyRotationRetry) + expect(error).toBeInstanceOf(Error) + expect(error.reason).toBe("quota_limit") + expect(error.name).toBe("KeyRotationRetry") + }) +}) diff --git a/packages/opencode/test/provider/key-rotator.test.ts b/packages/opencode/test/provider/key-rotator.test.ts new file mode 100644 index 000000000000..22c50929cce6 --- /dev/null +++ b/packages/opencode/test/provider/key-rotator.test.ts @@ -0,0 +1,138 @@ +import { afterEach, describe, expect, it, mock } from "bun:test" +import { KeyRotator, parseKeys } from "../../src/provider/key-rotator" + +const mockIsThrottled = mock(async (_source: string, _key: string) => false) +const mockAddThrottle = mock(async (_source: string, _key: string, _dur: number) => {}) + +const store = { isThrottled: mockIsThrottled, addThrottle: mockAddThrottle } + +afterEach(() => { + mockIsThrottled.mockReset() + mockAddThrottle.mockReset() + mockIsThrottled.mockImplementation(async () => false) + delete process.env.OPENCODE_API_KEY + delete process.env.OPENCODE_THROTTLE_ENABLE + delete process.env.OPENCODE_THROTTLE_DURATION +}) + +describe("parseKeys", () => { + it("returns empty array when env var not set", () => { + delete process.env.OPENCODE_API_KEY + expect(parseKeys()).toEqual([]) + }) + + it("returns single key", () => { + process.env.OPENCODE_API_KEY = "sk-abc" + expect(parseKeys()).toEqual(["sk-abc"]) + }) + + it("returns multiple keys split by comma", () => { + process.env.OPENCODE_API_KEY = "sk-abc,sk-def,sk-ghi" + expect(parseKeys()).toEqual(["sk-abc", "sk-def", "sk-ghi"]) + }) + + it("trims whitespace around commas", () => { + process.env.OPENCODE_API_KEY = "sk-abc , sk-def" + expect(parseKeys()).toEqual(["sk-abc", "sk-def"]) + }) + + it("drops empty strings after split", () => { + process.env.OPENCODE_API_KEY = "sk-abc,,sk-def" + expect(parseKeys()).toEqual(["sk-abc", "sk-def"]) + }) +}) + +describe("KeyRotator.parseKeys", () => { + it("exposes parseKeys function on the KeyRotator namespace", () => { + expect(KeyRotator.parseKeys).toBe(parseKeys) + process.env.OPENCODE_API_KEY = "sk-abc,sk-def" + expect(KeyRotator.parseKeys()).toEqual(["sk-abc", "sk-def"]) + }) +}) + +describe("KeyRotator.selectKey", () => { + it("returns first key when none are throttled or invalid", async () => { + const r = new KeyRotator(["key1", "key2"], store) + expect(await r.selectKey()).toBe("key1") + }) + + it("skips throttled key and returns next", async () => { + mockIsThrottled.mockImplementation(async (_s, key) => key === "key1") + const r = new KeyRotator(["key1", "key2"], store) + expect(await r.selectKey()).toBe("key2") + }) + + it("skips invalid key and returns next", async () => { + const r = new KeyRotator(["key1", "key2"], store) + r.markInvalid("key1") + expect(await r.selectKey()).toBe("key2") + }) + + it("returns null when all keys are throttled", async () => { + mockIsThrottled.mockImplementation(async () => true) + const r = new KeyRotator(["key1", "key2"], store) + expect(await r.selectKey()).toBeNull() + }) + + it("returns null when all keys are invalid", async () => { + const r = new KeyRotator(["key1", "key2"], store) + r.markInvalid("key1") + r.markInvalid("key2") + expect(await r.selectKey()).toBeNull() + }) + + it("skips throttle check when OPENCODE_THROTTLE_ENABLE=false", async () => { + process.env.OPENCODE_THROTTLE_ENABLE = "false" + mockIsThrottled.mockImplementation(async () => true) // would throttle if checked + const r = new KeyRotator(["key1"], store) + expect(await r.selectKey()).toBe("key1") // not skipped because throttle disabled + expect(mockIsThrottled).not.toHaveBeenCalled() + }) +}) + +describe("KeyRotator.recordThrottle", () => { + it("calls ThrottleStore.addThrottle with correct source and default duration", async () => { + const r = new KeyRotator(["key1"], store) + await r.recordThrottle("key1") + expect(mockAddThrottle).toHaveBeenCalledWith("OPENCODE_API_KEY", "key1", 120) + }) + + it("uses OPENCODE_THROTTLE_DURATION when set", async () => { + process.env.OPENCODE_THROTTLE_DURATION = "60" + const r = new KeyRotator(["key1"], store) + await r.recordThrottle("key1") + expect(mockAddThrottle).toHaveBeenCalledWith("OPENCODE_API_KEY", "key1", 60) + }) +}) + +describe("KeyRotator.markInvalid", () => { + it("causes the key to be skipped in selectKey", async () => { + const r = new KeyRotator(["key1", "key2"], store) + r.markInvalid("key1") + expect(await r.selectKey()).toBe("key2") + }) + + it("does not write to throttle.json", async () => { + const r = new KeyRotator(["key1"], store) + r.markInvalid("key1") + expect(mockAddThrottle).not.toHaveBeenCalled() + }) +}) + +describe("KeyRotator.hasAlternative", () => { + it("returns true when other non-invalid keys exist", () => { + const r = new KeyRotator(["key1", "key2", "key3"], store) + expect(r.hasAlternative("key1")).toBe(true) + }) + + it("returns false when only one key and it is current", () => { + const r = new KeyRotator(["key1"], store) + expect(r.hasAlternative("key1")).toBe(false) + }) + + it("returns false when other keys are all invalid", () => { + const r = new KeyRotator(["key1", "key2"], store) + r.markInvalid("key2") + expect(r.hasAlternative("key1")).toBe(false) + }) +}) diff --git a/packages/opencode/test/provider/rotation-logger.test.ts b/packages/opencode/test/provider/rotation-logger.test.ts new file mode 100644 index 000000000000..fbb8c7363682 --- /dev/null +++ b/packages/opencode/test/provider/rotation-logger.test.ts @@ -0,0 +1,70 @@ +import { afterEach, beforeEach, describe, expect, it } from "bun:test" +import path from "path" +import fs from "fs/promises" +import os from "os" +import { createRotationLogger } from "../../src/provider/rotation-logger" + +const tmpDir = path.join(os.tmpdir(), "opencode-welan-test-" + process.pid) +const configDir = path.join(tmpDir, "xdg", "opencode") +const logDir = path.join(tmpDir, "log") +const welanFile = path.join(configDir, "welan-log.txt") +const opencodeLogFile = path.join(logDir, "opencode.log") +const rotationLogger = createRotationLogger({ configDir, logDir }) + +beforeEach(async () => { + await fs.mkdir(configDir, { recursive: true }) + await fs.mkdir(logDir, { recursive: true }) +}) + +afterEach(async () => { + await fs.rm(tmpDir, { recursive: true, force: true }) + delete process.env.OPENCODE_WELAN_LOG + delete process.env.OPENCODE_PRINT_LOGS +}) + +function sleep(ms: number) { + return new Promise((r) => setTimeout(r, ms)) +} + +describe("RotationLogger.log", () => { + it("appends a line to welan.txt with correct format", async () => { + rotationLogger.log("info", "test message") + await sleep(50) // fire-and-forget: wait for write + const content = await fs.readFile(welanFile, "utf8") + expect(content).toMatch(/\[.*\] \[.*\] \[INFO\] test message\n/) + }) + + it("appends multiple lines in order", async () => { + rotationLogger.log("info", "first") + rotationLogger.log("warn", "second") + await sleep(50) + const content = await fs.readFile(welanFile, "utf8") + const lines = content.trim().split("\n") + expect(lines).toHaveLength(2) + expect(lines[0]).toContain("[INFO] first") + expect(lines[1]).toContain("[WARN] second") + // Both lines share the same PID prefix (first column) + const prefix = lines[0].split("] [")[0].slice(1) + expect(lines[1].startsWith(`[${prefix}]`)).toBe(true) + }) + + it("also writes to the regular opencode log format", async () => { + rotationLogger.log("warn", "regular log message") + await sleep(50) + const content = await fs.readFile(opencodeLogFile, "utf8") + expect(content).toContain("level=WARN") + expect(content).toMatch(/run=[^\s]+/) + expect(content).toContain('message="key-rotation: regular log message"') + }) + + it("does nothing when OPENCODE_WELAN_LOG=false", async () => { + process.env.OPENCODE_WELAN_LOG = "false" + rotationLogger.log("info", "should not appear") + await sleep(50) + const exists = await fs + .stat(welanFile) + .then(() => true) + .catch(() => false) + expect(exists).toBe(false) + }) +}) diff --git a/packages/opencode/test/provider/throttle-store.test.ts b/packages/opencode/test/provider/throttle-store.test.ts new file mode 100644 index 000000000000..1bb1ae8390f8 --- /dev/null +++ b/packages/opencode/test/provider/throttle-store.test.ts @@ -0,0 +1,110 @@ +import { afterEach, beforeEach, describe, expect, it } from "bun:test" +import path from "path" +import fs from "fs/promises" +import os from "os" +import { createThrottleStore } from "../../src/provider/throttle-store" + +const tmpDir = path.join(os.tmpdir(), "opencode-throttle-test-" + process.pid) +const configDir = path.join(tmpDir, "xdg", "opencode") +const throttleFile = path.join(configDir, "throttle.json") +const throttleStore = createThrottleStore({ configDir }) + +beforeEach(async () => { + await fs.mkdir(configDir, { recursive: true }) +}) + +afterEach(async () => { + await fs.rm(tmpDir, { recursive: true, force: true }) +}) + +describe("ThrottleStore.isThrottled", () => { + it("returns false when throttle.json does not exist", async () => { + expect(await throttleStore.isThrottled("OPENCODE_API_KEY", "key1")).toBe(false) + }) + + it("returns true when key has an active throttle record", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([{ source: "OPENCODE_API_KEY", key: "key1", startTime: now - 1000, endTime: now + 60_000 }]), + ) + expect(await throttleStore.isThrottled("OPENCODE_API_KEY", "key1")).toBe(true) + }) + + it("returns false when key record is expired", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([{ source: "OPENCODE_API_KEY", key: "key1", startTime: now - 120_000, endTime: now - 1000 }]), + ) + expect(await throttleStore.isThrottled("OPENCODE_API_KEY", "key1")).toBe(false) + }) + + it("returns false for a different key not in the file", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([{ source: "OPENCODE_API_KEY", key: "key1", startTime: now - 1000, endTime: now + 60_000 }]), + ) + expect(await throttleStore.isThrottled("OPENCODE_API_KEY", "key2")).toBe(false) + }) + + it("returns false on JSON parse error (file is corrupt)", async () => { + await fs.writeFile(throttleFile, "not-json") + expect(await throttleStore.isThrottled("OPENCODE_API_KEY", "key1")).toBe(false) + }) +}) + +describe("ThrottleStore.addThrottle", () => { + it("creates throttle.json with a new record", async () => { + await throttleStore.addThrottle("OPENCODE_API_KEY", "key1", 120) + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(1) + expect(data[0].source).toBe("OPENCODE_API_KEY") + expect(data[0].key).toBe("key1") + expect(data[0].endTime - data[0].startTime).toBe(120 * 60 * 1000) + }) + + it("upserts: updates existing record for same key", async () => { + await throttleStore.addThrottle("OPENCODE_API_KEY", "key1", 60) + await throttleStore.addThrottle("OPENCODE_API_KEY", "key1", 120) + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(1) + expect(data[0].endTime - data[0].startTime).toBe(120 * 60 * 1000) + }) + + it("appends: keeps other keys when adding a new one", async () => { + await throttleStore.addThrottle("OPENCODE_API_KEY", "key1", 60) + await throttleStore.addThrottle("OPENCODE_API_KEY", "key2", 120) + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(2) + }) +}) + +describe("ThrottleStore.cleanExpired", () => { + it("removes expired record for the given key", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([ + { source: "OPENCODE_API_KEY", key: "key1", startTime: now - 120_000, endTime: now - 1000 }, + { source: "OPENCODE_API_KEY", key: "key2", startTime: now - 1000, endTime: now + 60_000 }, + ]), + ) + await throttleStore.cleanExpired("OPENCODE_API_KEY", "key1") + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(1) + expect(data[0].key).toBe("key2") + }) + + it("does nothing when key is not in the file", async () => { + const now = Date.now() + await fs.writeFile( + throttleFile, + JSON.stringify([{ source: "OPENCODE_API_KEY", key: "key2", startTime: now, endTime: now + 60_000 }]), + ) + await throttleStore.cleanExpired("OPENCODE_API_KEY", "key1") + const data = JSON.parse(await fs.readFile(throttleFile, "utf8")) + expect(data).toHaveLength(1) + }) +}) diff --git a/packages/opencode/test/session/retry.test.ts b/packages/opencode/test/session/retry.test.ts index 506853a08a63..4fe72a94c0bb 100644 --- a/packages/opencode/test/session/retry.test.ts +++ b/packages/opencode/test/session/retry.test.ts @@ -165,6 +165,52 @@ describe("session.retry quota limits", () => { expect(SessionRetry.isQuotaOrRateLimitAPIError(transient)).toBe(false) }) + test("isInvalidKeyAPIError matches serialized 401 APIError", () => { + const invalid = Schema.decodeUnknownSync(SessionV1.APIError.Schema)( + new SessionV1.APIError({ + message: "Unauthorized", + isRetryable: false, + statusCode: 401, + responseBody: JSON.stringify({ error: "Unauthorized" }), + }).toObject(), + ) + expect(SessionRetry.isInvalidKeyAPIError(invalid)).toBe(true) + }) + + test("isInvalidKeyAPIError matches APIError produced from 401 APICallError", () => { + const apicall = new APICallError({ + message: "Unauthorized", + url: "https://api.example.com/v1/chat/completions", + requestBodyValues: {}, + statusCode: 401, + responseHeaders: { "content-type": "application/json" }, + responseBody: JSON.stringify({ error: "Unauthorized" }), + isRetryable: false, + }) + const normalized = MessageV2.fromError(apicall, { providerID }) + expect(SessionRetry.isInvalidKeyAPIError(normalized)).toBe(true) + }) + + test("isInvalidKeyAPIError ignores non-auth API errors", () => { + const rateLimit = Schema.decodeUnknownSync(SessionV1.APIError.Schema)( + new SessionV1.APIError({ + message: "Rate limit exceeded", + isRetryable: true, + statusCode: 429, + }).toObject(), + ) + expect(SessionRetry.isInvalidKeyAPIError(rateLimit)).toBe(false) + + const badRequest = Schema.decodeUnknownSync(SessionV1.APIError.Schema)( + new SessionV1.APIError({ + message: "Bad request", + isRetryable: false, + statusCode: 400, + }).toObject(), + ) + expect(SessionRetry.isInvalidKeyAPIError(badRequest)).toBe(false) + }) + test("isQuotaOrRateLimitRetryStatus prefers action reasons over message text", () => { expect( SessionRetry.isQuotaOrRateLimitRetryStatus({ diff --git a/packages/opencode/test/session/snapshot-tool-race.test.ts b/packages/opencode/test/session/snapshot-tool-race.test.ts index 1265237840f3..2ec7673b426a 100644 --- a/packages/opencode/test/session/snapshot-tool-race.test.ts +++ b/packages/opencode/test/session/snapshot-tool-race.test.ts @@ -137,10 +137,10 @@ it.live("tool execution produces non-empty session diff (snapshot race)", () => // Use bash tool (always registered) to create a file const command = `echo 'snapshot race test content' > ${path.join(dir, "race-test.txt")}` - yield* llm.toolMatch((hit) => JSON.stringify(hit.body).includes("create the file"), "bash", { + yield* llm.tool("bash", { command, }) - yield* llm.textMatch((hit) => JSON.stringify(hit.body).includes("bash"), "done") + yield* llm.text("done") // Seed user message yield* prompt.prompt({ @@ -186,4 +186,5 @@ it.live("tool execution produces non-empty session diff (snapshot race)", () => }), { git: true, config: providerCfg }, ), + 15_000, )