Orphaned Gate.serve() REPLs are never reaped — multi-GB leak under headless server

[1-Bart-1]

Summary

Gate worker processes started via Gate.serve() register a session with the server but are never terminated when abandoned. Over many editor/agent sessions and projects they accumulate as idle multi-GB Julia processes. On my machine I found four orphaned serve REPLs (one project had two) plus a leaked /tmp/kaimon_test_runner child — ~20 GB of RSS held by processes that had been idle for hours to over a day.

Root cause

1. close_session! (src/session.jl) only sets session.state = CLOSED — it never signals the OS process, so the worker keeps running.
2. The only stale-session reaper (_reap_stale_sessions! in src/tui/view.jl) is driven from the TUI render loop and operates on MCP client sessions (STANDALONE_SESSIONS), not on the gate worker processes (ConnectionManager REPLConnections with .pid). A headless -m Kaimon server never ticks it.
3. Idle can't be measured from last_seen: health pings flow through _req_send_recv, which refreshes last_seen every cycle. There was no activity timestamp independent of pings.

Net effect: nothing ever sends a signal to an abandoned gate, even though REPLConnection already carries everything needed (pid, project_path, status, allow_restart).

Proposed fix

Branch: 1-Bart-1:fix/reap-orphaned-serve-repls · commit 7a8ec78

Reap idle gates from the ConnectionManager health loop (runs in both TUI and headless modes):

• Add a ping-independent last_tool_call timestamp, stamped via _note_tool_activity! at every real tool call / eval.
• _should_reap_idle_gate flags a :connected, restartable, non-paused gate idle beyond a threshold (evaluating / stalled / debug-paused / allow_restart=false extension gates are skipped).
• Such gates are shut down gracefully via send_shutdown! and pruned through the existing removal path.
• Threshold is a new gate_idle_reap_seconds preference, default 0 (disabled) so behaviour is opt-in.

Happy to open this as a PR.

[kahliburke]

@1-Bart-1 Thank you for the bug report, I will take a look. I am working on a 2.0.0 release and I may have fixed a related issue but I will make sure to incorporate any additional fixes in your branch.

[kahliburke]

Thanks for the detailed report and the branch, @1-Bart-1. I looked into this for the 2.0 release.

When I instrumented it, Kaimon-spawned sessions (start_session and agent sessions) don't actually orphan. I tested both exit paths on a headless server:

• Clean restart: the spawned gate is reaped by stop_all_sessions!.
• kill -9 of the server: the spawned gate still dies — it's a PTY child, so it gets SIGHUP when the parent goes away.

A process check after each confirmed no spawned gate survived. So the persistent multi-GB Gate.serve() REPLs are user/editor-started gates — separate processes Kaimon connects to but doesn't own.

That persistence is intended. Sessions are meant to survive a Kaimon restart so REPL state isn't lost; a restarted Kaimon (or an agent) finds them again via socket discovery / ping and reuses them. So Kaimon doesn't force-kill gates it didn't spawn — one might hold work you still want. An opt-in idle cleanup scoped to spawned sessions only is something I'd consider as a future feature.

One item in your report is a real leak and I'll fix it: the /tmp/kaimon_test_runner child. The test-runner subprocess isn't tied into shutdown and can orphan if a run hangs or Kaimon exits mid-run, and its temp script isn't cleaned up. I'll track in-flight runs, reap them on shutdown, and remove the temp file.

Your report also prompted a wider change: the maintenance you noted was TUI-only (stale-session reaping, extension/session monitoring) now runs headless too in 2.0, along with headless analytics-DB init and periodic indexing.

One question: were the four orphaned serve REPLs started by start_session / the VS Code extension, or by hand (julia> Gate.serve())? I'll revisit this after the 2.0 release — if you can share more detail on how those REPLs were started and your setup, it'll help me track down whether there's a real issue in Kaimon's path.

Thanks again.

[1-Bart-1]

I start the REPLs by hand (julia> Gate.serve()). Then using claude code to connect to the MCP server.

[1-Bart-1]

Thanks for looking into this, and explaining this. Makes sense.

[kahliburke]

Following up now that 2.0 is further along, and given your confirmation that all four orphaned REPLs were hand-started (julia> Gate.serve()) rather than via start_session / the VS Code extension.

Decision: not taking the idle-reaper approach from the branch as-is.

The proposed fix reaps idle gates from the ConnectionManager health loop based on a gate_idle_reap_seconds threshold. The problem is it can't distinguish "abandoned" from "idle but wanted" for gates Kaimon doesn't own. A hand-started Gate.serve() REPL is a process Kaimon merely connects to — it may hold expensive in-memory state (loaded data, a half-finished computation) you want to keep across a Kaimon restart. That cross-restart persistence is intentional: a restarted Kaimon (or a fresh agent) rediscovers these gates via socket discovery / ping and reuses them. Time-based force-reaping would kill exactly the long-lived REPLs people deliberately keep running.

What did change for 2.0:

• Kaimon-spawned sessions (start_session, agent sessions) are confirmed not to orphan — reaped by stop_all_sessions! on clean stop, and they die on kill -9 of the server too (PTY children get SIGHUP).
• The maintenance that was TUI-only (stale-session reaping, session/extension monitoring) now runs headless as well.

So for the user-started case there isn't a Kaimon-side leak to fix per se — but I don't want to just close this, because the underlying ask ("don't let abandoned hand-started REPLs pile up") is real. Before adding anything I'd want to investigate:

• An opt-in, spawned-or-explicitly-tagged-only idle cleanup that never force-kills a gate Kaimon didn't start (or that wasn't declared reapable), so the safe default is preserved.
• A ping-independent activity signal — the health ping refreshes last_seen, so idleness can't be measured from it today. The branch's last_tool_call idea is reasonable here.
• Whether a lighter-weight / lower-memory idle mode for serve gates is a better lever than reaping at all (overlaps with the memory-footprint discussion in Excessive memory consumption #33).

Keeping this open as an investigation item. Thanks again @1-Bart-1 — the report and branch were a useful prompt even though the fix will likely land differently.