You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f6c86bb
Browse filesBrowse the repository at this point in the historyBrowse files
@@ -30,7 +37,7 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
30
37
|`src/parser_ts_types.ts`| Indexer and Worker (Critical Path) | Minimal structural typings for the node-tree-sitter API surface we touch | TsPoint, TsNode, TsTree, TsParser, TsParserCtor |
31
38
|`src/parser_types.ts`| Indexer and Worker (Critical Path) | Type definitions shared between the tree-sitter parser (Layer 7) and the layers that query the index | SymbolEntry, RefEntry, FileIndexEntry, Language, TREE_SITTER_LANGUAGES |
32
39
|`src/reconcile.ts`| Indexer and Worker (Critical Path) | Catch-up reconciliation: detect index drift caused by edits token-goat never saw | DEFAULT_RECONCILE_BUDGET_MS, PreHashStamp, readPreHashStamp, restampVerifiedUnchanged, ReconcileOptions |
33
-
|`src/worker.ts`| Indexer and Worker (Critical Path) | Background worker: drain the dirty queue and re-index changed files |clearRetryCount, cleanupWorkerStateFiles, pendingEmbeddings, embedPolicyResolver, makeIndexer|
40
+
|`src/worker.ts`| Indexer and Worker (Critical Path) | Background worker: drain the dirty queue and re-index changed files |FOREGROUND_ACTIVITY_YIELD_MS, hasRecentForegroundActivity, clearRetryCount, cleanupWorkerStateFiles, pendingEmbeddings|
34
41
|`src/constants.ts`| Storage and Database | Package-wide constants: version re-export, data-directory resolution, and the canonical TOKEN_GOAT_*| VERSION, dataDirForHome, dataDir, ensureDataDirPrivate, tokenGoatHome |
35
42
|`src/db.ts`| Storage and Database | SQLite connection management and index-DB schema | FTS_TOKENIZER, SCHEMA_VERSION, enableWalWithRetry, WAL_SIZE_LIMIT_BYTES, WAL_SIZE_LIMIT_PRAGMA |
36
43
|`src/db_integrity.ts`| Storage and Database | Integrity probe and quarantine for the global index database | QuickCheckResult, QuarantineResult, quickCheckDb, quarantineMalformedDb, recreateDb |
@@ -286,11 +293,13 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
286
293
|`src/render/types.ts`| Hints, Guidance, and Formatting | Data-transfer types for the stats renderer | Sparklines, TotalStats, KindStat, DayStat, ProjectStat |
287
294
|`src/cli_doctor.ts`| Diagnostics, Doctor, and Maintenance | Doctor CLI helpers — diagnose token-goat health | autoReclaimRow, repairBlocker, runDoctorRepair, DoctorRepairResult, checkInstall |
288
295
|`src/cli_doctor_guidance.ts`| Diagnostics, Doctor, and Maintenance | Diagnostics, Doctor, and Maintenance module: cli_doctor_guidance.ts | INSTRUCTION_GATE_BEGIN, INSTRUCTION_GATE_END, CODEX_GATE_BEGIN, CODEX_GATE_END, InstructionGateCandidate |
296
+
|`src/cli_doctor_harness_perf.ts`| Diagnostics, Doctor, and Maintenance | Diagnostic checks for AI harness session storage bloat and AV/EDR exclusion posture | DEFAULT_MAX_HARNESS_SESSION_BYTES, MAX_SESSION_SCAN_FILES, AV_EXCLUSION_DISCLAIMER, HarnessDirectoryCandidate, getHarnessStorageCandidates |
289
297
|`src/cli_doctor_hooks.ts`| Diagnostics, Doctor, and Maintenance | Harness hook repairs for `token-goat doctor --repair` and `token-goat install`| shimIsCurrent, wiringPointsAtMissingFile, HarnessRepairResult, repairHarnessHooks |
290
298
|`src/cli_doctor_index.ts`| Diagnostics, Doctor, and Maintenance | Index and reindex-queue diagnostics for token-goat doctor | DB_SIZE_WARN_BYTES, freelistBytes, CategoryByteShare, dbCategoryBreakdown, oversizeDbMessage |
291
299
|`src/cli_doctor_native.ts`| Diagnostics, Doctor, and Maintenance |`doctor`'s native hook client rows: per harness and scope, which form of hook command is wired | NativeWiring, NativeAvailability, nativeAvailability, nativeWirings, formatNativeCounts |
292
300
|`src/cli_doctor_platforms.ts`| Diagnostics, Doctor, and Maintenance | Platform and harness integration diagnostics for token-goat doctor | globalMcpConfigPath, checkGlobalMcpConfig, VSCODE_USER_SCOPE_MIGRATED_NOTE, VSCODE_PROJECT_SCOPE_COVERAGE_NOTE, VSCODE_USER_SCOPE_MULTIROOT_NOTE |
293
301
|`src/cli_doctor_process.ts`| Diagnostics, Doctor, and Maintenance | Process table and MCP process health diagnostics for token-goat doctor | ProcessInfo, ProcessListFailure, isTokenGoatResidentProcess, checkMcpProcessHealth, processListOutput |
302
+
|`src/cli_doctor_repair.ts`| Diagnostics, Doctor, and Maintenance | Repair utilities and routines for token-goat doctor --fix / --repair | DoctorRepairResult, autoReclaimRow, repairBlocker, runDoctorRepair |
294
303
|`src/cli_doctor_security.ts`| Diagnostics, Doctor, and Maintenance | Security posture and configuration override diagnostics for token-goat doctor | LOCKED_BOOLEAN_SAFE_VALUE, lockedEnvOverridableKeys, EnvOverriddenSetting, envOverriddenSecuritySettings, dataDirPermissionResult |
295
304
|`src/pack.ts`| Diagnostics, Doctor, and Maintenance | Diagnostics, Doctor, and Maintenance module: pack.ts | PackFile, PackResult, SecretHit, BudgetEntry, BudgetResult |
296
305
|`src/disk_cache.ts`| Caches and Output Stores | Cross-process disk persistence for id-keyed content caches | DEFAULT_MAX_COUNT, DEFAULT_MAX_AGE_MS, tokenGoatHome, blobPath, isBlobStale |
@@ -312,6 +321,8 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
312
321
|`src/image_isobmff.ts`| Media, Documents, and Structured Formats | Header-only probe for ISO base media (ISOBMFF) still images: AVIF and HEIC/HEIF | IsoBmffMeta, probeIsoBmffMeta |
313
322
|`src/image_ocr.ts`| Media, Documents, and Structured Formats | OCR — extract text from text-heavy images (screenshots of a terminal, a stack trace, a table, an editor, a doc page) | DEFAULT_OCR_LANG, SUPPORTED_OCR_LANGS, getOcrLangSpec, isSupportedOcrLang, resolveOcrLangs |
|`src/json_query.ts`| Media, Documents, and Structured Formats | Narrow structural summary + path-based extraction for `token-goat json-outline` / `json-query`, so a multi-thousand-line | JsonValueType, jsonType, JsonFieldSummary, JsonOutlineArray, JsonOutlineObject |
316
327
|`src/pdf_extract.ts`| Media, Documents, and Structured Formats | PDF -> plain text extraction for `token-goat pdf-extract`, so a PDF's useful content reaches the model as text instead | PdfExtractResult, MAX_PDF_TEXT_BYTES, MAX_PDF_INPUT_BYTES, MAX_PDF_TEXT_ITEMS, MAX_OUTLINE_ENTRIES |
317
328
|`src/sqlite_driver.ts`| Media, Documents, and Structured Formats | A better-sqlite3-shaped facade over Node's built-in `node:sqlite`| suppressSqliteExperimentalWarning, sqliteResultCodeName, SqliteColumn, SqliteRunResult, SqliteStatement |
@@ -551,17 +562,6 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
// Stale session-snapshot sweep runs on the same loop as the dirty-queue drain (see runWorkerLoop) but throttled to this interval -- cleanup_stale's own default 24h staleness window doesn't need finer-grained sweeping than hourly, and a full directory scan on every 2s poll tick would be wasteful.
37
38
constSNAPSHOT_CLEANUP_INTERVAL_MS=60*60*1000
38
39
40
+
/** How long a foreground session write suppresses heavy background worker maintenance (embedding backlog walks, known-roots sweeps, snapshot/cache sweeps, idle vacuuming) so interactive CLI turns do not compete with background I/O. */
41
+
exportconstFOREGROUND_ACTIVITY_YIELD_MS=6_000
42
+
43
+
/** True when any session state in `sessDir` (defaults to {@link sessionsDir}) was updated within {@link FOREGROUND_ACTIVITY_YIELD_MS} of `now`. Absent or unreadable directories return false so idle daemons and isolated tests proceed normally. */
/** How many {@link drainOnce} cycles to skip between opportunistic {@link pruneDeletedFiles} sweeps. A `git mv`, directory rename, `git checkout <branch>`, or `git clean` never fires an Edit hook for the paths it touches, so a plain dirty-queue drain (which only reconciles a deletion when the exact old path was enqueued) never notices those rows are gone and they orphan in the index forever. `pruneDeletedFiles` walks every indexed path under a project root and stats each one to catch this — too expensive to repeat on every ~2s poll tick, so it only runs every Nth cycle (mirrors SNAPSHOT_CLEANUP_INTERVAL_MS's throttling above, keyed on drain-cycle count here instead of wall time since the trigger is "enough drain activity", not "enough time elapsed"). */
40
54
constPRUNE_EVERY_N_DRAINS=30
41
55
@@ -875,11 +889,12 @@ export async function runWorkerLoop(
875
889
// The sweep never throws; a request it left claimed is swept again next cycle.
876
890
}
877
891
if(retiringNow())break
892
+
constforegroundBusy=hasRecentForegroundActivity()
878
893
if(processed>0){
879
894
lastActiveMs=Date.now()
880
895
lastBusyMs=lastActiveMs
881
-
}elseif(inFlightEmbeddings.size===0){
882
-
// Only with the queue drained and every embed settled, so the backlog never competes with an edit. See requeueStaleEmbeddings.
// Only with the queue drained, every embed settled, and no active foreground hook turn, so the backlog never competes with an edit or interactive CLI turn. See requeueStaleEmbeddings.
883
898
try{
884
899
constready=embeddingsReady(dir)
885
900
conststamps=disabledStampCount()
@@ -896,8 +911,8 @@ export async function runWorkerLoop(
896
911
// An unreadable index this cycle: the cursor stays put and the next idle cycle tries again.
897
912
}
898
913
}
899
-
// Sweep stale session-snapshot directories on the same periodic loop as the dirty-queue drain above, so accumulating session_snapshots/<sessionId>/ dirs get cleaned up on a schedule instead of growing unbounded for the life of the daemon.
// Sweep stale session-snapshot directories on the same periodic loop as the dirty-queue drain above, so accumulating session_snapshots/<sessionId>/ dirs get cleaned up on a schedule instead of growing unbounded for the life of the daemon. Deferred while a foreground CLI turn is active so synchronous directory scans do not compete with interactive tool calls.
@@ -920,7 +935,7 @@ export async function runWorkerLoop(
920
935
lastSnapshotCleanupMs=Date.now()
921
936
}
922
937
// Auto-prune dead file rows across every known project root -- see sweepKnownRoots' docstring for the safety guarantees (grace period before treating an unreachable root as gone, anomaly-ratio guard against a mount-point outage under a live root). This is what keeps the shared global.db from silently accumulating dead rows indefinitely the way it used to, when pruning only ever ran via a manually-invoked `token-goat index`.
@@ -933,7 +948,7 @@ export async function runWorkerLoop(
933
948
}
934
949
// Perform off-peak idle space reclamation / vacuum on global.db when the daemon has been continuously idle (no dirty items) for at least IDLE_VACUUM_COOLDOWN_MS and at least IDLE_VACUUM_INTERVAL_MS has elapsed since the last vacuum.
// No database yet is nothing to reclaim, not a refusal: it neither counts nor logs, and the mark stays put so the next cycle looks again. idleReclaim keeps its own false for a missing file, which tests/wal_size_bounded.test.ts pins.
0 commit comments