Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions README.md

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low

### Mutating (29)

The README labels the mutating tool table as Mutating (29), but removing resync_agents leaves only 28 listed tools, so the documented count is incorrect. Update the heading to Mutating (28) and adjust any aggregate counts that are intended to describe the listed tools.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @README.md around line 165:

The README labels the mutating tool table as `Mutating (29)`, but removing `resync_agents` leaves only 28 listed tools, so the documented count is incorrect. Update the heading to `Mutating (28)` and adjust any aggregate counts that are intended to describe the listed tools.

Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ The other 30 definitions, including `interact`, are interim ToolSearch-deferred

**Terminal control** — `list_surfaces` `control_health` `select_workspace` `create_workspace` `delete_workspace` `new_split` `new_surface` `move_surface` `send_input` `send_command` `send_key` `read_screen` `rename_tab` `close_surface` `browser_surface`

**Agent lifecycle** — `spawn_agent` `new_worktree_split` `spawn_in_workspace` `resync_agents` `send_to` `send_to_agent` `wait_for` `wait_for_all` `interact` `stop_agent` `kill` `supersede_agent_goal` `broadcast`
**Agent lifecycle** — `spawn_agent` `new_worktree_split` `spawn_in_workspace` `send_to` `send_to_agent` `wait_for` `wait_for_all` `interact` `stop_agent` `kill` `supersede_agent_goal` `broadcast`

**Metacomm (agent inbox)** — `dispatch_to_agent` `inbox_check`

Expand Down Expand Up @@ -183,7 +183,6 @@ The other 30 definitions, including `interact`, are interim ToolSearch-deferred
| `spawn_agent` | Spawn a CLI agent and return an `agent_id` for routing |
| `new_worktree_split` | Deprecated one-release alias; use `spawn_agent(worktree:true, placement:"worker")` |
| `spawn_in_workspace` | Deprecated one-release alias; create/reuse a workspace and call `spawn_agent` for each managed agent |
| `resync_agents` | Re-sync the agent registry from live surfaces |
| `dispatch_to_agent` | Append a task to an agent's inbox file (deterministic write channel) |
| `send_to` | Send by agent ID or raw surface using `mode:"agent"|"surface"|"command"|"key"` |
| `send_to_agent` | Deprecated one-release alias for `send_to(mode:"agent")` |
Expand Down
21 changes: 21 additions & 0 deletions docs/control-plane-invariants.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,27 @@ recovery, and sidebar/reporting decisions.
An empty/failed surface listing is inconclusive (`unknown`); only a non-empty topology lacking a
specific surface proves absence; stale records reap on the next non-empty scan.

### Bounded eviction windows (#480)

A registry row is dropped once its absence is confirmed for a window, and the window depends on
whether a live observer claims the row. Neither window is unbounded, which is the invariant the
measured 36-day ghosts violated.

| Row | Window | Constant | Path |
| --- | --- | --- | --- |
| `surface_observer_id` equals the current observer | 5 s of continuous absence | `SURFACE_EVICTION_CONFIRMATION_MS` (`src/agent-registry.ts`) | sweep + `list_agents` |
| `surface_observer_id` is null or from a prior observer generation | 60 s of continuous absence, where absence means no live surface bears the row's identity key — its UUID when it has one, else its ref | `UNCLAIMED_SURFACE_EVICTION_CONFIRMATION_MS` (`src/agent-registry.ts`) | sweep + `list_agents` |

Ownership stops one observer mutating another's *live* row; it is not a claim on a row that no
live surface bears. An unclaimed row is evicted, never crash-marked: eviction is the reversible
direction, because `listMerged` re-mints a row from discovery if the pane turns out to be alive.

One exception, and it is deliberate: a row whose captured `cli_session_id` still resolves to a
session artifact on disk is retained regardless of the window. `resumeAgent` has no ownership gate,
so that row is the record resume-by-ID acts on, and the registry is the only `agent_id` →
`cli_session_id` mapping. `missing` and session-less rows still evict, so the ghost class #480 was
filed about still closes.

## Allowed Transitions

Allowed transitions are intentionally narrower than current ad hoc state movement:
Expand Down
26 changes: 25 additions & 1 deletion src/agent-facade.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ import {
rawResumeNeedsCwd,
rawResumeSupported,
} from "./agent-command.js";
import {
resumeArtifactStatus,
type ResumeArtifactStatus,
} from "./resume-verification.js";

export type AgentStatePayload = AgentRecord & {
resumable: boolean;
Expand Down Expand Up @@ -57,6 +61,19 @@ export function resumeInvocationForAgent(
if (!record.cli_session_id) {
return { command: null, reason: "no CLI session has been captured" };
}
// #482: a formattable id is not a resumable agent. A seat that survived a
// restart keeps the OLD id, so the command would open a fresh session
// wearing the seat's name. Refuse on proof of absence only -- an
// unverifiable store leaves the claim standing.
if (resumeArtifactStatus(record.cli, record.cli_session_id) === "missing") {
return {
command: null,
reason:
`captured ${record.cli} session ${record.cli_session_id} is not in ` +
`the harness session store; resuming it would start a NEW session ` +
`under this agent's name, not restore it`,
};
}
const cwd = resumeCwdForAgent(record);
if (!record.launcher_name) {
if (!cwd && rawResumeNeedsCwd(record.cli)) {
Expand Down Expand Up @@ -147,6 +164,13 @@ export function toObservedPublicAgent(
const registryObservedAtMs = derivedAtMs;
const resumeCommand = resumeCommandForAgent(record);
const resumable = !!resumeCommand;
// #482 provenance: `disk` means a session artifact was looked for and
// found (or proven absent). `registry` means the claim is unverified.
const artifactStatus: ResumeArtifactStatus = record.cli_session_id
? resumeArtifactStatus(record.cli, record.cli_session_id)
: "unverifiable";
const resumableSource: ObservationSource =
artifactStatus === "unverifiable" ? "registry" : "disk";
const hasScreenModelObservation =
opts.screenObservedAtMs !== undefined && opts.screenModel != null;
const model = hasScreenModelObservation
Expand Down Expand Up @@ -178,7 +202,7 @@ export function toObservedPublicAgent(
"registry",
registryObservedAtMs,
),
resumable: observed(resumable, "registry", registryObservedAtMs),
resumable: observed(resumable, resumableSource, registryObservedAtMs),
submit_verified: observed(
record.submit_verified ?? null,
"registry",
Expand Down
135 changes: 128 additions & 7 deletions src/agent-registry.ts

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium

if (liveSurfaceKeys.has(surfaceKey)) {

evictSurfaceless deletes a managed record after the 60-second unclaimed window even when its surface_id ref is still live under a different UUID, causing the record to lose its managed metadata and be re-minted as an auto record. isUnclaimedAbsenceConfirmed checks only agentSurfaceKey(agent), which is UUID-only for UUID-bearing records; reset the timer when either the record's ref or UUID is present.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/agent-registry.ts around line 1956:

`evictSurfaceless` deletes a managed record after the 60-second unclaimed window even when its `surface_id` ref is still live under a different UUID, causing the record to lose its managed metadata and be re-minted as an auto record. `isUnclaimedAbsenceConfirmed` checks only `agentSurfaceKey(agent)`, which is UUID-only for UUID-bearing records; reset the timer when either the record's ref or UUID is present.

Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ import {
import { validateSurfaceIdentityBijection } from "./surface-topology.js";
import { deriveCmuxObserverOwnerId } from "./cmux-observer-identity.js";
import { inferRepoFromDirectory } from "./repo-workspace.js";
import { resumeArtifactStatus } from "./resume-verification.js";

export type SurfaceProvider = () => Promise<CmuxSurface[]>;

Expand Down Expand Up @@ -105,6 +106,23 @@ export function deriveSurfaceObserverId(

export const SURFACE_EVICTION_CONFIRMATION_MS = 5_000;

/**
* Absence window for rows NO live observer claims (#480).
*
* `canMutateForObservedAbsence` requires an exact observer match, so a row
* whose `surface_observer_id` is null (pre-observer-identity) or belongs to a
* dead socket generation could never be evicted, never be crash-marked, and
* never be purged: worst-case survival was unbounded. Measured 2026-08-19:
* four such rows, oldest 36 days, `list_agents` 17 vs `list_surfaces` 13.
*
* Ownership exists to stop one observer mutating another's LIVE row. It is not
* a claim on a row that no live surface bears — matched on the row's UUID when
* it has one, else on its ref — across a continuous absence window. This constant is that window: twelve consecutive
* 5 s sweeps of proven absence before an unclaimed row is dropped, so the
* worst case is bounded and documented rather than infinite.
*/
export const UNCLAIMED_SURFACE_EVICTION_CONFIRMATION_MS = 60_000;

export interface AgentFilter {
state?: AgentState;
repo?: string;
Expand Down Expand Up @@ -501,6 +519,16 @@ export class AgentRegistry {
string,
{ surfaceId: string; firstObservedAt: number }
>();
/**
* Absence clock for rows this observer does not own (#480). Kept separate
* from `surfacelessObservations` on purpose: that map is cleared by the
* ownership gate itself (`isSurfacelessConfirmed`), so an unclaimed row can
* never accumulate time in it.
*/
private unclaimedAbsenceObservations = new Map<
string,
{ surfaceId: string; firstObservedAt: number }
>();
private stateMgr: StateManager;
private surfaceProvider: SurfaceProvider;
private observerId: string | null;
Expand Down Expand Up @@ -645,7 +673,7 @@ export class AgentRegistry {
async reconstitute(opts: SurfaceAbsenceOptions = {}): Promise<Set<string>> {
this.agents.clear();
this.aliases.clear();
this.surfacelessObservations.clear();
this.clearAbsenceObservations();

const stateFiles = this.stateMgr.listStates();
for (const record of stateFiles) {
Expand Down Expand Up @@ -749,7 +777,7 @@ export class AgentRegistry {
// Incomplete or contradictory identity evidence can prove neither
// presence nor absence.
// Reset pending absence timers so a later valid scan starts fresh.
this.surfacelessObservations.clear();
this.clearAbsenceObservations();
return new Set();
}
const liveSurfaceKeys = this.liveSurfaceKeys(surfaces);
Expand Down Expand Up @@ -909,6 +937,7 @@ export class AgentRegistry {
const aliases = this.aliasesResolvingTo(resolved);
this.agents.delete(resolved);
this.surfacelessObservations.delete(resolved);
this.unclaimedAbsenceObservations.delete(resolved);
this.aliases.delete(agentId);
this.aliases.delete(resolved);
for (const alias of aliases) {
Expand Down Expand Up @@ -1013,7 +1042,7 @@ export class AgentRegistry {
if (!discoveryIsBijective || discoveryHasMixedIdentity) {
// A degraded discovery scan must break any pending negative-evidence
// streak even when exact UUID matches remain safe for positive sync.
this.surfacelessObservations.clear();
this.clearAbsenceObservations();
}
if (!discoveryIsBijective) {
return this.list(opts?.filter).map((record) => ({
Expand Down Expand Up @@ -1237,7 +1266,7 @@ export class AgentRegistry {
const discoveryHasMixedIdentity =
hasMixedDiscoveryIdentityCoverage(discovered);
if (!discoveryIsBijective || discoveryHasMixedIdentity) {
this.surfacelessObservations.clear();
this.clearAbsenceObservations();
}
if (!discoveryIsBijective) {
return opts?.agentId ? this.get(opts.agentId) : null;
Expand Down Expand Up @@ -1629,6 +1658,12 @@ export class AgentRegistry {
if (surfacelessObservation?.surfaceId === this.agentSurfaceKey(record)) {
this.surfacelessObservations.set(newAgentId, surfacelessObservation);
}
const unclaimedObservation =
this.unclaimedAbsenceObservations.get(oldAgentId);
this.unclaimedAbsenceObservations.delete(oldAgentId);
if (unclaimedObservation?.surfaceId === this.agentSurfaceKey(record)) {
this.unclaimedAbsenceObservations.set(newAgentId, unclaimedObservation);
}
for (const [alias, target] of this.aliases) {
if (target === oldAgentId) {
this.aliases.set(alias, newAgentId);
Expand Down Expand Up @@ -1693,7 +1728,7 @@ export class AgentRegistry {
return [];
}
if (!hasCoherentSurfaceIdentity(surfaces)) {
this.surfacelessObservations.clear();
this.clearAbsenceObservations();
return [];
}

Expand All @@ -1704,10 +1739,40 @@ export class AgentRegistry {
for (const [id, agent] of [...this.agents.entries()]) {
if (agent.transcript_session_capture_deferred === true) {
this.surfacelessObservations.delete(agent.agent_id);
this.unclaimedAbsenceObservations.delete(agent.agent_id);
continue;
}
if (this.matchingLiveSurface(agent, surfaces)) {
this.surfacelessObservations.delete(agent.agent_id);
this.unclaimedAbsenceObservations.delete(agent.agent_id);
continue;
}
// #480: rows no live observer claims take the bounded unclaimed path
// instead of dying at the ownership gate below. They cannot be evicted,
// crash-marked (`reconcileSurfaces` applies the same gate) or recovered
// (`recoverCrashedAgents` quarantines unowned rows) on any other path,
// so without this they live forever.
if (!this.canMutateForObservedAbsence(agent, observerSnapshot.ownerId)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High src/agent-registry.ts:1759

evictSurfaceless deletes a still-live agent record owned by another observer after the local observer has failed to see its surface for 60 seconds. Because surfaceProvider() only reports the current observer's topology, this absence is not global evidence; retain the ownership gate unless a cross-observer absence proof is available.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/agent-registry.ts around line 1759:

`evictSurfaceless` deletes a still-live agent record owned by another observer after the local observer has failed to see its surface for 60 seconds. Because `surfaceProvider()` only reports the current observer's topology, this absence is not global evidence; retain the ownership gate unless a cross-observer absence proof is available.

// The row is the only `agent_id` -> `cli_session_id` mapping, and
// `resumeAgent` is the one recovery path with no ownership gate. So a
// row whose captured session is still on disk is not a ghost: it is
// the record resume-by-ID acts on, and a successful resume re-stamps
// it with this observer. Deleting it would strand a live transcript.
// Only a PRESENT artifact retains: `missing` restores nothing, and
// `unverifiable` (no store on this machine) must not make eviction
// depend on a directory's existence -- that would reopen #480 wherever
// the harness store is absent.
if (this.hasVerifiedResumeArtifact(agent)) {
this.unclaimedAbsenceObservations.delete(agent.agent_id);
continue;
}
if (!this.isUnclaimedAbsenceConfirmed(agent, opts)) {
continue;
}
const removedUnclaimedId = this.evictUnchecked(id);
if (removedUnclaimedId) {
evicted.push(removedUnclaimedId);
}
continue;
}
if (!this.isSurfaceAbsenceAuthoritative(agent, surfaces)) {
Expand Down Expand Up @@ -1874,6 +1939,52 @@ export class AgentRegistry {
return now - observation.firstObservedAt >= confirmationMs;
}

/**
* #480/#482: a captured session this machine can still see on disk — the
* one thing an unclaimed row still protects, since `resumeAgent` has no
* ownership gate and the registry holds the only agent_id -> session map.
*/
private hasVerifiedResumeArtifact(agent: AgentRecord): boolean {
if (!agent.cli_session_id) return false;
return resumeArtifactStatus(agent.cli, agent.cli_session_id) === "present";
}

/**
* Continuous absence of a row that this observer does not own (#480).
*
* The row's identity is ONE key: its UUID when it has one, else its ref
* (`agentSurfaceKey`). Absence means the caller's `matchingLiveSurface`
* found nothing for that key in a coherent, non-empty scan.
*
* Deliberately does NOT consult `isSurfaceAbsenceAuthoritative`: that helper
* refuses to read a UUID-less row's absence in a UUID-bearing topology,
* because a live occupant sitting ON the same mutable ref proves nothing
* about the row. Here the ref is not occupied at all. That is real absence
* evidence, and it is the only evidence an unclaimed row can ever produce.
*
* Eviction, not crash-marking: dropping the registry row is the reversible
* direction. If the pane were somehow alive, `listMerged` re-mints it from
* discovery on the next call; marking a live agent `error` would not
* self-correct.
*/
private isUnclaimedAbsenceConfirmed(
agent: AgentRecord,
opts: { now?: number },
): boolean {
const surfaceKey = this.agentSurfaceKey(agent);
const confirmationMs = UNCLAIMED_SURFACE_EVICTION_CONFIRMATION_MS;
const now = opts.now ?? Date.now();
const observation = this.unclaimedAbsenceObservations.get(agent.agent_id);
if (!observation || observation.surfaceId !== surfaceKey) {
this.unclaimedAbsenceObservations.set(agent.agent_id, {
surfaceId: surfaceKey,
firstObservedAt: now,
});
return false;
}
return now - observation.firstObservedAt >= confirmationMs;
}

private canMutateForObservedAbsence(
agent: AgentRecord,
observerEpoch?: string | null,
Expand All @@ -1895,6 +2006,11 @@ export class AgentRegistry {
return !owner || Boolean(observerId && owner === observerId);
}

private clearAbsenceObservations(): void {
this.surfacelessObservations.clear();
this.unclaimedAbsenceObservations.clear();
}

private clearSurfacelessObservationsForLiveSurfaces(
liveSurfaceKeys: ReadonlySet<string>,
): void {
Expand All @@ -1903,6 +2019,11 @@ export class AgentRegistry {
this.surfacelessObservations.delete(agentId);
}
}
for (const [agentId, observation] of this.unclaimedAbsenceObservations) {
if (liveSurfaceKeys.has(observation.surfaceId)) {
this.unclaimedAbsenceObservations.delete(agentId);
}
}
}

repairFromDiscovery(
Expand All @@ -1916,7 +2037,7 @@ export class AgentRegistry {
!hasBijectiveDiscoveryIdentity(discovered) ||
hasMixedDiscoveryIdentityCoverage(discovered)
) {
this.surfacelessObservations.clear();
this.clearAbsenceObservations();
return { repaired: [], evicted: [], skipped: [] };
}
const repaired: RegistryRepairEntry[] = [];
Expand Down Expand Up @@ -2463,7 +2584,7 @@ export class AgentRegistry {
return 0;
}
if (!hasCoherentSurfaceIdentity(surfaces)) {
this.surfacelessObservations.clear();
this.clearAbsenceObservations();
return 0;
}
const liveSurfaceKeys = this.liveSurfaceKeys(surfaces);
Expand Down
4 changes: 3 additions & 1 deletion src/agent-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ export type AgentFunction = "implementor" | "reviewer" | "gatherer";
export type AgentPlacement = "left" | "right";
export type SurfaceProvenance = "cmuxlayer_spawn" | "unknown";
export type SeatIdentityStatus = "ok" | "mismatch" | "unknown";
export type ObservationSource = "screen" | "registry" | "process";
// `disk` is a filesystem observation (today: the harness session artifact
// behind `resumable`, #482), as opposed to a remembered registry field.
export type ObservationSource = "screen" | "registry" | "process" | "disk";
export type AgentReviveOutcome =
"pending" | "failed" | "revived" | "unrecoverable";
export type AgentHaltType =
Expand Down
17 changes: 0 additions & 17 deletions src/format.ts
Original file line number Diff line number Diff line change
Expand Up @@ -307,20 +307,3 @@ export function formatDelivery(
submit = " \u00b7 submit_verified=null (not attempted)";
return `\u2714 ${action} \u2500 ${head}${submit}`;
}

export function formatResync(diff: {
added: string[];
evicted: string[];
repaired?: unknown[];
reflowed?: unknown[];
mismatches: string[];
orphaned?: string[];
}): string {
const added = diff.added.length;
const evicted = diff.evicted.length;
const repaired = diff.repaired?.length ?? 0;
const reflowed = diff.reflowed?.length ?? 0;
const mismatches = diff.mismatches.length;
const orphaned = diff.orphaned?.length ?? 0;
return `✔ resync_agents — added: ${added} repaired: ${repaired} reflowed: ${reflowed} evicted: ${evicted} mismatches: ${mismatches} orphaned: ${orphaned}`;
}
Loading
Loading