Skip to content

Commit f6c86bb

Browse files
committed
perf(worker): yield background backlog and housekeeping during active foreground turns
1 parent 50fd1fe commit f6c86bb

11 files changed

Lines changed: 141 additions & 27 deletions

‎CLAUDE.arch.md‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,15 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
1111
| --- | --- | --- | --- |
1212
| `src/cli.ts` | Entry and CLI Dispatch | Entry and CLI Dispatch module: cli.ts | expandGlobs, CliError, formatCommandError, out, err |
1313
| `src/cli_cmd_analysis.ts` | Entry and CLI Dispatch | CLI command registration for code analysis, dependency, graph, and git commands | GuardFn, registerAnalysisCommands |
14+
| `src/cli_cmd_content.ts` | Entry and CLI Dispatch | Entry and CLI Dispatch module: cli_cmd_content.ts | registerContentCommands |
15+
| `src/cli_cmd_diagnostics.ts` | Entry and CLI Dispatch | Entry and CLI Dispatch module: cli_cmd_diagnostics.ts | registerDiagnosticCommands |
16+
| `src/cli_cmd_documents.ts` | Entry and CLI Dispatch | Entry and CLI Dispatch module: cli_cmd_documents.ts | registerDocumentCommands |
1417
| `src/cli_cmd_formats.ts` | Entry and CLI Dispatch | CLI command registration for file formats: PDF, Office, Structured Data, SQLite, Media, and Text utilities | GuardFn, fenceOcrText, cmdImageMeta, cmdImageText, cmdImageSlice |
18+
| `src/cli_cmd_index.ts` | Entry and CLI Dispatch | Entry and CLI Dispatch module: cli_cmd_index.ts | registerIndexCommands |
19+
| `src/cli_cmd_inspect.ts` | Entry and CLI Dispatch | Entry and CLI Dispatch module: cli_cmd_inspect.ts | registerInspectCommands |
20+
| `src/cli_cmd_integrations.ts` | Entry and CLI Dispatch | Entry and CLI Dispatch module: cli_cmd_integrations.ts | registerIntegrationCommands |
1521
| `src/cli_cmd_session.ts` | Entry and CLI Dispatch | CLI command registration for session, caching, audit, skill, and note commands | GuardFn, registerSessionCommands |
22+
| `src/cli_cmd_system.ts` | Entry and CLI Dispatch | Entry and CLI Dispatch module: cli_cmd_system.ts | registerSystemCommands |
1623
| `src/cli_dispatch.ts` | Entry and CLI Dispatch | Shared CLI execution and dispatch helpers | readStdinPaths, runExit, runExitText, ExtraArgsNoteOpts, noteExtraFileArgs |
1724
| `src/cli_help.ts` | Entry and CLI Dispatch | Compact CLI help generation | generateCompactHelp |
1825
| `src/main.ts` | Entry and CLI Dispatch | Package executable | — |
@@ -30,7 +37,7 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
3037
| `src/parser_ts_types.ts` | Indexer and Worker (Critical Path) | Minimal structural typings for the node-tree-sitter API surface we touch | TsPoint, TsNode, TsTree, TsParser, TsParserCtor |
3138
| `src/parser_types.ts` | Indexer and Worker (Critical Path) | Type definitions shared between the tree-sitter parser (Layer 7) and the layers that query the index | SymbolEntry, RefEntry, FileIndexEntry, Language, TREE_SITTER_LANGUAGES |
3239
| `src/reconcile.ts` | Indexer and Worker (Critical Path) | Catch-up reconciliation: detect index drift caused by edits token-goat never saw | DEFAULT_RECONCILE_BUDGET_MS, PreHashStamp, readPreHashStamp, restampVerifiedUnchanged, ReconcileOptions |
33-
| `src/worker.ts` | Indexer and Worker (Critical Path) | Background worker: drain the dirty queue and re-index changed files | clearRetryCount, cleanupWorkerStateFiles, pendingEmbeddings, embedPolicyResolver, makeIndexer |
40+
| `src/worker.ts` | Indexer and Worker (Critical Path) | Background worker: drain the dirty queue and re-index changed files | FOREGROUND_ACTIVITY_YIELD_MS, hasRecentForegroundActivity, clearRetryCount, cleanupWorkerStateFiles, pendingEmbeddings |
3441
| `src/constants.ts` | Storage and Database | Package-wide constants: version re-export, data-directory resolution, and the canonical TOKEN_GOAT_* | VERSION, dataDirForHome, dataDir, ensureDataDirPrivate, tokenGoatHome |
3542
| `src/db.ts` | Storage and Database | SQLite connection management and index-DB schema | FTS_TOKENIZER, SCHEMA_VERSION, enableWalWithRetry, WAL_SIZE_LIMIT_BYTES, WAL_SIZE_LIMIT_PRAGMA |
3643
| `src/db_integrity.ts` | Storage and Database | Integrity probe and quarantine for the global index database | QuickCheckResult, QuarantineResult, quickCheckDb, quarantineMalformedDb, recreateDb |
@@ -286,11 +293,13 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
286293
| `src/render/types.ts` | Hints, Guidance, and Formatting | Data-transfer types for the stats renderer | Sparklines, TotalStats, KindStat, DayStat, ProjectStat |
287294
| `src/cli_doctor.ts` | Diagnostics, Doctor, and Maintenance | Doctor CLI helpers — diagnose token-goat health | autoReclaimRow, repairBlocker, runDoctorRepair, DoctorRepairResult, checkInstall |
288295
| `src/cli_doctor_guidance.ts` | Diagnostics, Doctor, and Maintenance | Diagnostics, Doctor, and Maintenance module: cli_doctor_guidance.ts | INSTRUCTION_GATE_BEGIN, INSTRUCTION_GATE_END, CODEX_GATE_BEGIN, CODEX_GATE_END, InstructionGateCandidate |
296+
| `src/cli_doctor_harness_perf.ts` | Diagnostics, Doctor, and Maintenance | Diagnostic checks for AI harness session storage bloat and AV/EDR exclusion posture | DEFAULT_MAX_HARNESS_SESSION_BYTES, MAX_SESSION_SCAN_FILES, AV_EXCLUSION_DISCLAIMER, HarnessDirectoryCandidate, getHarnessStorageCandidates |
289297
| `src/cli_doctor_hooks.ts` | Diagnostics, Doctor, and Maintenance | Harness hook repairs for `token-goat doctor --repair` and `token-goat install` | shimIsCurrent, wiringPointsAtMissingFile, HarnessRepairResult, repairHarnessHooks |
290298
| `src/cli_doctor_index.ts` | Diagnostics, Doctor, and Maintenance | Index and reindex-queue diagnostics for token-goat doctor | DB_SIZE_WARN_BYTES, freelistBytes, CategoryByteShare, dbCategoryBreakdown, oversizeDbMessage |
291299
| `src/cli_doctor_native.ts` | Diagnostics, Doctor, and Maintenance | `doctor`'s native hook client rows: per harness and scope, which form of hook command is wired | NativeWiring, NativeAvailability, nativeAvailability, nativeWirings, formatNativeCounts |
292300
| `src/cli_doctor_platforms.ts` | Diagnostics, Doctor, and Maintenance | Platform and harness integration diagnostics for token-goat doctor | globalMcpConfigPath, checkGlobalMcpConfig, VSCODE_USER_SCOPE_MIGRATED_NOTE, VSCODE_PROJECT_SCOPE_COVERAGE_NOTE, VSCODE_USER_SCOPE_MULTIROOT_NOTE |
293301
| `src/cli_doctor_process.ts` | Diagnostics, Doctor, and Maintenance | Process table and MCP process health diagnostics for token-goat doctor | ProcessInfo, ProcessListFailure, isTokenGoatResidentProcess, checkMcpProcessHealth, processListOutput |
302+
| `src/cli_doctor_repair.ts` | Diagnostics, Doctor, and Maintenance | Repair utilities and routines for token-goat doctor --fix / --repair | DoctorRepairResult, autoReclaimRow, repairBlocker, runDoctorRepair |
294303
| `src/cli_doctor_security.ts` | Diagnostics, Doctor, and Maintenance | Security posture and configuration override diagnostics for token-goat doctor | LOCKED_BOOLEAN_SAFE_VALUE, lockedEnvOverridableKeys, EnvOverriddenSetting, envOverriddenSecuritySettings, dataDirPermissionResult |
295304
| `src/pack.ts` | Diagnostics, Doctor, and Maintenance | Diagnostics, Doctor, and Maintenance module: pack.ts | PackFile, PackResult, SecretHit, BudgetEntry, BudgetResult |
296305
| `src/disk_cache.ts` | Caches and Output Stores | Cross-process disk persistence for id-keyed content caches | DEFAULT_MAX_COUNT, DEFAULT_MAX_AGE_MS, tokenGoatHome, blobPath, isBlobStale |
@@ -312,6 +321,8 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
312321
| `src/image_isobmff.ts` | Media, Documents, and Structured Formats | Header-only probe for ISO base media (ISOBMFF) still images: AVIF and HEIC/HEIF | IsoBmffMeta, probeIsoBmffMeta |
313322
| `src/image_ocr.ts` | Media, Documents, and Structured Formats | OCR — extract text from text-heavy images (screenshots of a terminal, a stack trace, a table, an editor, a doc page) | DEFAULT_OCR_LANG, SUPPORTED_OCR_LANGS, getOcrLangSpec, isSupportedOcrLang, resolveOcrLangs |
314323
| `src/image_slice.ts` | Media, Documents, and Structured Formats | Image slicing, difference calculation, and compact visual layout inspection | ImageSliceOptions, ImageSliceResult, ImageDiffResult, sliceImage, diffImages |
324+
| `src/json_path_eval.ts` | Media, Documents, and Structured Formats | Media, Documents, and Structured Formats module: json_path_eval.ts | JsonValueType, jsonType, MAX_RECURSIVE_NODES, MAX_RECURSIVE_DEPTH, JsonQueryResult |
325+
| `src/json_path_parser.ts` | Media, Documents, and Structured Formats | Media, Documents, and Structured Formats module: json_path_parser.ts | ProjectionObjectField, FilterOperator, FilterLiteral, FilterCondition, FilterNode |
315326
| `src/json_query.ts` | Media, Documents, and Structured Formats | Narrow structural summary + path-based extraction for `token-goat json-outline` / `json-query`, so a multi-thousand-line | JsonValueType, jsonType, JsonFieldSummary, JsonOutlineArray, JsonOutlineObject |
316327
| `src/pdf_extract.ts` | Media, Documents, and Structured Formats | PDF -> plain text extraction for `token-goat pdf-extract`, so a PDF's useful content reaches the model as text instead | PdfExtractResult, MAX_PDF_TEXT_BYTES, MAX_PDF_INPUT_BYTES, MAX_PDF_TEXT_ITEMS, MAX_OUTLINE_ENTRIES |
317328
| `src/sqlite_driver.ts` | Media, Documents, and Structured Formats | A better-sqlite3-shaped facade over Node's built-in `node:sqlite` | suppressSqliteExperimentalWarning, sqliteResultCodeName, SqliteColumn, SqliteRunResult, SqliteStatement |
@@ -551,17 +562,6 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
551562
| `src/xlsx_extract.ts` | Specialized Subsystems | Excel (.xlsx) narrow-slice reader | XlsxScanTooLargeError, SheetInfo, listSheets, headSheet, allSheetsHeadText |
552563
| `src/xlsx_reader.ts` | Specialized Subsystems | Minimal in-house SpreadsheetML (.xlsx) reader | ExcelCell, ExcelRow, ExcelWorksheet, ExcelWorkbook, refToColumn |
553564
| `src/zip_bounds.ts` | Specialized Subsystems | Shared decompression bounds for zip-format archives | MAX_ZIP_INPUT_BYTES, MAX_ZIP_OUTPUT_BYTES, ZipStreamModule, ZipOutputTooLargeError, ZipInputTooLargeError |
554-
| `src/cli_cmd_content.ts` | Uncategorized | Uncategorized module: cli_cmd_content.ts | registerContentCommands |
555-
| `src/cli_cmd_diagnostics.ts` | Uncategorized | Uncategorized module: cli_cmd_diagnostics.ts | registerDiagnosticCommands |
556-
| `src/cli_cmd_documents.ts` | Uncategorized | Uncategorized module: cli_cmd_documents.ts | registerDocumentCommands |
557-
| `src/cli_cmd_index.ts` | Uncategorized | Uncategorized module: cli_cmd_index.ts | registerIndexCommands |
558-
| `src/cli_cmd_inspect.ts` | Uncategorized | Uncategorized module: cli_cmd_inspect.ts | registerInspectCommands |
559-
| `src/cli_cmd_integrations.ts` | Uncategorized | Uncategorized module: cli_cmd_integrations.ts | registerIntegrationCommands |
560-
| `src/cli_cmd_system.ts` | Uncategorized | Uncategorized module: cli_cmd_system.ts | registerSystemCommands |
561-
| `src/cli_doctor_harness_perf.ts` | Uncategorized | Diagnostic checks for AI harness session storage bloat and AV/EDR exclusion posture | DEFAULT_MAX_HARNESS_SESSION_BYTES, MAX_SESSION_SCAN_FILES, AV_EXCLUSION_DISCLAIMER, HarnessDirectoryCandidate, getHarnessStorageCandidates |
562-
| `src/cli_doctor_repair.ts` | Uncategorized | Repair utilities and routines for token-goat doctor --fix / --repair | DoctorRepairResult, autoReclaimRow, repairBlocker, runDoctorRepair |
563-
| `src/json_path_eval.ts` | Uncategorized | Uncategorized module: json_path_eval.ts | JsonValueType, jsonType, MAX_RECURSIVE_NODES, MAX_RECURSIVE_DEPTH, JsonQueryResult |
564-
| `src/json_path_parser.ts` | Uncategorized | Uncategorized module: json_path_parser.ts | ProjectionObjectField, FilterOperator, FilterLiteral, FilterCondition, FilterNode |
565565
<!-- ARCH_COMPONENTS_END -->
566566
## Storage Layout
567567

‎config/arch-docs.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,15 @@
1010
"layers": [
1111
["src/cli.ts","Entry and CLI Dispatch"],
1212
["src/cli_cmd_analysis.ts","Entry and CLI Dispatch"],
13+
["src/cli_cmd_content.ts","Entry and CLI Dispatch"],
14+
["src/cli_cmd_diagnostics.ts","Entry and CLI Dispatch"],
15+
["src/cli_cmd_documents.ts","Entry and CLI Dispatch"],
1316
["src/cli_cmd_formats.ts","Entry and CLI Dispatch"],
17+
["src/cli_cmd_index.ts","Entry and CLI Dispatch"],
18+
["src/cli_cmd_inspect.ts","Entry and CLI Dispatch"],
19+
["src/cli_cmd_integrations.ts","Entry and CLI Dispatch"],
1420
["src/cli_cmd_session.ts","Entry and CLI Dispatch"],
21+
["src/cli_cmd_system.ts","Entry and CLI Dispatch"],
1522
["src/cli_dispatch.ts","Entry and CLI Dispatch"],
1623
["src/cli_help.ts","Entry and CLI Dispatch"],
1724
["src/main.ts","Entry and CLI Dispatch"],
@@ -141,11 +148,13 @@
141148
["src/render","Hints, Guidance, and Formatting"],
142149
["src/cli_doctor.ts","Diagnostics, Doctor, and Maintenance"],
143150
["src/cli_doctor_guidance.ts","Diagnostics, Doctor, and Maintenance"],
151+
["src/cli_doctor_harness_perf.ts","Diagnostics, Doctor, and Maintenance"],
144152
["src/cli_doctor_hooks.ts","Diagnostics, Doctor, and Maintenance"],
145153
["src/cli_doctor_index.ts","Diagnostics, Doctor, and Maintenance"],
146154
["src/cli_doctor_native.ts","Diagnostics, Doctor, and Maintenance"],
147155
["src/cli_doctor_platforms.ts","Diagnostics, Doctor, and Maintenance"],
148156
["src/cli_doctor_process.ts","Diagnostics, Doctor, and Maintenance"],
157+
["src/cli_doctor_repair.ts","Diagnostics, Doctor, and Maintenance"],
149158
["src/cli_doctor_security.ts","Diagnostics, Doctor, and Maintenance"],
150159
["src/pack.ts","Diagnostics, Doctor, and Maintenance"],
151160
["src/disk_cache.ts","Caches and Output Stores"],
@@ -167,6 +176,8 @@
167176
["src/image_isobmff.ts","Media, Documents, and Structured Formats"],
168177
["src/image_ocr.ts","Media, Documents, and Structured Formats"],
169178
["src/image_slice.ts","Media, Documents, and Structured Formats"],
179+
["src/json_path_eval.ts","Media, Documents, and Structured Formats"],
180+
["src/json_path_parser.ts","Media, Documents, and Structured Formats"],
170181
["src/json_query.ts","Media, Documents, and Structured Formats"],
171182
["src/pdf_extract.ts","Media, Documents, and Structured Formats"],
172183
["src/sqlite_driver.ts","Media, Documents, and Structured Formats"],

‎src/worker.ts‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import { reclaimIndex, walCheckpointBusyNoWait } from './index_reclaim.js'
2323
import { applyIndexingPriority } from './process_priority.js'
2424
import { cleanup_stale } from './snapshots.js'
2525
import { sweepCacheRoots } from './disk_cache.js'
26+
import { sessionsDir } from './sessions_dir.js'
2627
import { sweepStaleShrinkCopies } from './shrink_temp_copies.js'
2728
import { findProject, isUnderSystemTemp } from './project.js'
2829
import { registerReset } from './reset.js'
@@ -36,6 +37,19 @@ import { appendWorkerErrorLog, claimCurrentWorker, readPidFile, resolvePollInter
3637
// Stale session-snapshot sweep runs on the same loop as the dirty-queue drain (see runWorkerLoop) but throttled to this interval -- cleanup_stale's own default 24h staleness window doesn't need finer-grained sweeping than hourly, and a full directory scan on every 2s poll tick would be wasteful.
3738
const SNAPSHOT_CLEANUP_INTERVAL_MS = 60 * 60 * 1000
3839

40+
/** How long a foreground session write suppresses heavy background worker maintenance (embedding backlog walks, known-roots sweeps, snapshot/cache sweeps, idle vacuuming) so interactive CLI turns do not compete with background I/O. */
41+
export const FOREGROUND_ACTIVITY_YIELD_MS = 6_000
42+
43+
/** True when any session state in `sessDir` (defaults to {@link sessionsDir}) was updated within {@link FOREGROUND_ACTIVITY_YIELD_MS} of `now`. Absent or unreadable directories return false so idle daemons and isolated tests proceed normally. */
44+
export function hasRecentForegroundActivity(now = Date.now(), sessDir: string = sessionsDir()): boolean {
45+
try {
46+
const mtimeMs = fs.statSync(sessDir).mtimeMs
47+
return now - mtimeMs >= 0 && now - mtimeMs < FOREGROUND_ACTIVITY_YIELD_MS
48+
} catch {
49+
return false
50+
}
51+
}
52+
3953
/** How many {@link drainOnce} cycles to skip between opportunistic {@link pruneDeletedFiles} sweeps. A `git mv`, directory rename, `git checkout <branch>`, or `git clean` never fires an Edit hook for the paths it touches, so a plain dirty-queue drain (which only reconciles a deletion when the exact old path was enqueued) never notices those rows are gone and they orphan in the index forever. `pruneDeletedFiles` walks every indexed path under a project root and stats each one to catch this — too expensive to repeat on every ~2s poll tick, so it only runs every Nth cycle (mirrors SNAPSHOT_CLEANUP_INTERVAL_MS's throttling above, keyed on drain-cycle count here instead of wall time since the trigger is "enough drain activity", not "enough time elapsed"). */
4054
const PRUNE_EVERY_N_DRAINS = 30
4155

@@ -875,11 +889,12 @@ export async function runWorkerLoop(
875889
// The sweep never throws; a request it left claimed is swept again next cycle.
876890
}
877891
if (retiringNow()) break
892+
const foregroundBusy = hasRecentForegroundActivity()
878893
if (processed > 0) {
879894
lastActiveMs = Date.now()
880895
lastBusyMs = lastActiveMs
881-
} else if (inFlightEmbeddings.size === 0) {
882-
// Only with the queue drained and every embed settled, so the backlog never competes with an edit. See requeueStaleEmbeddings.
896+
} else if (inFlightEmbeddings.size === 0 && !foregroundBusy) {
897+
// Only with the queue drained, every embed settled, and no active foreground hook turn, so the backlog never competes with an edit or interactive CLI turn. See requeueStaleEmbeddings.
883898
try {
884899
const ready = embeddingsReady(dir)
885900
const stamps = disabledStampCount()
@@ -896,8 +911,8 @@ export async function runWorkerLoop(
896911
// An unreadable index this cycle: the cursor stays put and the next idle cycle tries again.
897912
}
898913
}
899-
// Sweep stale session-snapshot directories on the same periodic loop as the dirty-queue drain above, so accumulating session_snapshots/<sessionId>/ dirs get cleaned up on a schedule instead of growing unbounded for the life of the daemon.
900-
if (Date.now() - lastSnapshotCleanupMs >= SNAPSHOT_CLEANUP_INTERVAL_MS) {
914+
// Sweep stale session-snapshot directories on the same periodic loop as the dirty-queue drain above, so accumulating session_snapshots/<sessionId>/ dirs get cleaned up on a schedule instead of growing unbounded for the life of the daemon. Deferred while a foreground CLI turn is active so synchronous directory scans do not compete with interactive tool calls.
915+
if (!foregroundBusy && Date.now() - lastSnapshotCleanupMs >= SNAPSHOT_CLEANUP_INTERVAL_MS) {
901916
try {
902917
cleanup_stale()
903918
} catch {
@@ -920,7 +935,7 @@ export async function runWorkerLoop(
920935
lastSnapshotCleanupMs = Date.now()
921936
}
922937
// Auto-prune dead file rows across every known project root -- see sweepKnownRoots' docstring for the safety guarantees (grace period before treating an unreachable root as gone, anomaly-ratio guard against a mount-point outage under a live root). This is what keeps the shared global.db from silently accumulating dead rows indefinitely the way it used to, when pruning only ever ran via a manually-invoked `token-goat index`.
923-
if (Date.now() - lastKnownRootsSweepMs >= KNOWN_ROOTS_SWEEP_INTERVAL_MS) {
938+
if (!foregroundBusy && Date.now() - lastKnownRootsSweepMs >= KNOWN_ROOTS_SWEEP_INTERVAL_MS) {
924939
try {
925940
const result = sweepKnownRoots(path.join(dir, 'global.db'))
926941
if (result.flaggedRoots.length > 0) {
@@ -933,7 +948,7 @@ export async function runWorkerLoop(
933948
}
934949
// Perform off-peak idle space reclamation / vacuum on global.db when the daemon has been continuously idle (no dirty items) for at least IDLE_VACUUM_COOLDOWN_MS and at least IDLE_VACUUM_INTERVAL_MS has elapsed since the last vacuum.
935950
const now = Date.now()
936-
if (now - lastIdleVacuumMs >= IDLE_VACUUM_INTERVAL_MS && now - lastActiveMs >= IDLE_VACUUM_COOLDOWN_MS) {
951+
if (!foregroundBusy && now - lastIdleVacuumMs >= IDLE_VACUUM_INTERVAL_MS && now - lastActiveMs >= IDLE_VACUUM_COOLDOWN_MS) {
937952
const reclaimTarget = path.join(dir, 'global.db')
938953
// No database yet is nothing to reclaim, not a refusal: it neither counts nor logs, and the mark stays put so the next cycle looks again. idleReclaim keeps its own false for a missing file, which tests/wal_size_bounded.test.ts pins.
939954
if (fs.existsSync(reclaimTarget)) {

0 commit comments

Comments
 (0)