Skip to content

agent-runtime.md

Latest commit

 

History

History
230 lines (161 loc) · 50.4 KB

agent-runtime.md

File metadata and controls

230 lines (161 loc) · 50.4 KB
Raw

Agent runtime

The agent runtime is where an agent actually executes: the agent_runtime_bindings row that ties an agent principal to a driver_type and a workspace, the terminal (PTY) sessions the API Gateway spawns for direct chats, the headless CLI turns the pipeline spawns for group work, the command leases in session_registry, harness accounts (which CLI login a binding uses), hidden agent sessions, and the import of native Claude/Codex/Pi/Grok/OpenCode sessions into Choruz. A reader can use this page to add a driver, to debug why a terminal or headless turn will not start, or to understand binding, account and session state. Source: crates/choruz-agent-runtime/, crates/choruz-session/, crates/choruz-executor/, services/choruz-api-gateway/src/handlers_terminals.rs.

Owns

Path Role
crates/choruz-agent-runtime/src/binding.rs DriverType, BindingState, RuntimeBinding, native session discovery and normalize_workspace_path; no database client
crates/choruz-application/src/runtime_store/mod.rs RuntimeStore: binding creation, rebind, state and cursor updates, terminal anchors, capture metadata, and identity-checked native session writeback
crates/choruz-agent-runtime/src/headless.rs HeadlessDriver (Claude, Codex, Pi, Grok, OpenCode), from_driver_type, args, parse_output, validate_model, configure_command_workspace, harness_account_env
crates/choruz-application/src/runtime_store/policy.rs ConversationRuntimePolicy, AutoMode, UntaggedHumanMode, RuntimeStore::get_policy / upsert_policy over conversation_runtime_policies
crates/choruz-agent-runtime/src/session_catalog.rs HarnessKind, SessionCatalogScanner::from_env / scan, NativeSessionSummary, SessionScanResult
crates/choruz-session/src/store.rs PgSessionStore: session_registry epochs, agent_commands leases (assign_lease, assign_batch_leases, release_lease), list_runtime_status_for_agents, runtime-host commands (claim_runtime_host_command, heartbeat_runtime_host_command, complete_runtime_host_command)
crates/choruz-executor/src/wal.rs AdapterWal: turn records and incomplete-turn inspection
crates/choruz-host-runtime/src/headless.rs Device-local headless command construction, contained execution, timeout/cancellation and resume-error classification shared by pipeline and connector
services/choruz-pipeline/src/executor.rs Fenced turn orchestration, session_provenance_matches and stale-session cleanup using the shared headless owner (see message-pipeline)
crates/choruz-host-runtime/ Device-local operations shared by the gateway and the connector: HostRequest / execute (filesystem.rs, session scan, latest_native_session, the Codex home and session attribution in codex.rs), terminal.rs (TerminalSpec, TerminalSession, TerminalPool, ensure_terminal, terminal_cli_args, terminal_binary, is_terminal_driver)
crates/choruz-agent-runtime/src/process.rs ProcessContainer: child-process group containment shared by device execution and tool setup
crates/choruz-computer-use/ Device-local browser and desktop tool installation, enablement and diagnostics
services/choruz-api-gateway/src/host_runtime.rs, host_link.rs RuntimeHost: the device a binding runs on (for_binding reads config_json.runtime_host_id), call for a HostRequest, ensure_terminal / attach_terminal / write_terminal / resize_terminal / close_terminal; LocalHost serves the gateway's own device in-process, Linked sends the same requests over the device's host link (HostLinkHub, GET /v1/ws/runtime-hosts/link)
services/choruz-api-gateway/src/handlers_terminals.rs websocket_terminal, ensure_terminal, terminal_input, authorize_terminal_binding, the Codex capture orchestration (prepare_codex_spawn_if_needed, capture_codex_terminal_before_cleanup)
services/choruz-api-gateway/src/handlers_runtime.rs, handlers_runtime_status.rs Binding CRUD, validate_runtime_host, validate_harness_account, policies, reset_company_sessions, get_conversation_runtime_status
services/choruz-api-gateway/src/handlers_runtime_hosts.rs Harness account registration and verification by a runtime host; runtime-host command claim (routes registered by plugins/remote_control.rs)
services/choruz-api-gateway/src/handlers_harness_logins.rs Harness account sign-in: the company-facing start/read/callback routes, the in-process executor for accounts on the gateway's device, the host-facing claim/publish/complete routes the connector uses
crates/choruz-harness-login/ The Claude Code and Codex browser sign-in driver shared by the gateway and choruz-connector
services/choruz-api-gateway/src/handlers_workspace_sessions.rs scan_workspace_sessions, import_workspace_sessions, native_session_import_lock_key, ensure_outbox_helper
services/choruz-api-gateway/src/handlers_conversations.rs hide_agent_session, restore_hidden_agent_session
apps/web/lib/drivers/driver-availability.ts, apps/web/app/api/drivers/availability/route.ts, models/route.ts getDriverAvailability, resolveDriverBinary; GET /api/drivers/availability
apps/web/app/api/harness-accounts/ Web routes for listing, creating, probing, hiding and signing in harness accounts; POST /api/harness-accounts/default registers (and, on request, verifies) the login a device already has

Tables and migrations: agent_runtime_bindings, agent_turn_leases, conversation_runtime_policies (0003); driver_type allowlist (0004, 0013, 0021, 0026, 0027, 0028, 0029); one active binding per agent (0018); hybrid routing columns (0024); native_session_import (0032, V032, V033); conversation_hidden (V034); harness_account and the validate_runtime_binding_harness_account trigger (V035); harness_account_login (V036, V037); audit_log rows for binding changes (0001).

Data

DriverType (as_str) and where each runs:

driver_type Terminal (PTY on the binding's device via RuntimeHost) Headless (pipeline) Instruction file
claude_terminal yes yes (HeadlessDriver::Claude) CLAUDE.md
codex_terminal yes yes (Codex) AGENTS.md
muse_terminal yes yes (Muse) AGENTS.md
pi_terminal yes yes (Pi) AGENTS.md
grok_terminal yes yes (Grok) AGENTS.md
opencode_terminal yes yes (OpenCode) AGENTS.md
claude_print, codex_exec, codex_app_server no yes CLAUDE.md / AGENTS.md
acp no no (from_driver_type returns None) none
webhook_agent no outbox drain only; events leave through event_webhook and POST /v1/webhooks/flush CLAUDE.md

The constraint agent_runtime_bindings_driver_type_check allows the identities listed in the data model. Driver identity and execution are independent of new-agent eligibility: the plugin contract gates creation and import, not existing bindings. Muse uses the official CLI login and model configuration; isolated account management and historical import remain unavailable for Muse.

RuntimeBinding (binding.rs): id, conversation_id, agent_principal_id, driver_type, workspace_path, git_worktree_path, external_session_id, external_thread_id, last_event_cursor, last_acked_event_cursor, last_seen_server_seq, state (idle, running, paused, disabled, error; transitions guarded by BindingState::can_transition_to), last_error, in_flight_turn_id, last_trigger_message_id, config_json. Keys read from config_json: model, binary_path, runtime_host_id, harness_account_id, harness_account_name, harness_account_profile_kind (default or isolated), interaction_mode, external_session_mode, external_session_driver_type, external_session_provenance (process_captured or workspace_scan_verified), external_session_binding_id, terminal_session (TerminalSessionAnchor), terminal_capture (CodexTerminalCaptureMetadata), native_session_import, is_primary.

harness_account rows: company_id, optional runtime_host_id, driver_type in (claude_terminal, codex_terminal), name, profile_kind, account_fingerprint, subscription_type, status (pending, active, reauth_required, error, disabled), models_json, usage_json, probed_at. Each device and harness has one default account, the login the device already has; ensureDefaultHarnessAccount (apps/web/lib/agents/harness-accounts.ts) registers it the first time the Harness Accounts dialog or agent provisioning needs it, the dialog probes it on open through the gateway's probe route (probeHarnessAccount), and provisioning binds a Claude Code or Codex agent to it when no account was chosen (defaultHarnessAccountForLaunch; an unverified default leaves the binding without an account, so the agent inherits the device's login without quota data). company.multi_harness_accounts (V039, off by default) is the switch that lets the dialog add isolated sign-ins and Create Agent or Create Group choose among them. Hiding an account (DELETE, disabled_at) leaves the device's credentials and profile directory in place. Credentials stay in the device-local profile directory; harness_account_env resolves an isolated profile under CHORUZ_HARNESS_ACCOUNT_ROOT (default $HOME/.choruz/accounts) and returns the env var the CLI needs. harness_account_login rows carry the sign-in state machine (queued, awaiting_browser, authorizing, verified, failed, cancelled, expired) with authorization_url, user_code, callback_code, expires_at; runtime_host_id names the connector that runs the sign-in and is NULL when the API gateway runs it on its own device. A verified Harness identity makes the account active before the independent model and quota refresh finishes. A failed refresh preserves the active status and last complete snapshot, and does not advance probed_at.

native_session_import maps (workspace_path, driver_type, native_session_id) to one agent principal, one direct conversation and one binding. NativeSessionSummary (harness, native_session_id, title, workspace_path, updated_at, model, branch, archived) is what SessionCatalogScanner::scan returns; it reads $HOME, CODEX_HOME (default ~/.codex), PI_CODING_AGENT_SESSION_DIR (default ~/.pi/agent/sessions), GROK_HOME (default ~/.grok) and runs CHORUZ_OPENCODE_BINARY for OpenCode.

AgentRuntimeStatus (status = busy, queued or idle, active_command, queued_count, last_error) is computed from agent_commands by list_runtime_status_for_agents.

Binaries: terminals use config_json.binary_path or default_terminal_binary, overridden by CHORUZ_CLAUDE_BINARY, CHORUZ_CODEX_BINARY, CHORUZ_PI_BINARY, CHORUZ_GROK_BINARY, CHORUZ_OPENCODE_BINARY when the configured value is the bare default; headless turns use CHORUZ_*_CLI_PATH; getDriverAvailability checks the same CHORUZ_*_BINARY variables and reports available or unavailable with reason, setupHint and envVar.

Entry points

DELETE /v1/companies/{companyId}/harness-accounts/{accountId} records removal and stops managed processes on the account's device. The web account route forwards this operation to the gateway. HTTP 202 means cleanup remains pending; the account manager displays it until the background worker confirms completion. New bindings, logins and terminal input are denied while cleanup proceeds. Credentials and profile directories remain on the device. See account retirement.

Harness Accounts includes device-scoped Computer use setup. POST /v1/companies/{companyId}/computer-use reads diagnostics; supplying tool (browser or desktop) and enabled changes automatic skill provisioning and starts installation when enabled. Optional runtime_host_id routes to a linked device in the company. The gateway requires a human company member. Installation runs in the owning API/connector process independently of the panel, and is limited to upstream macOS/Linux installers. A process restart interrupts an installation; the operator can retry. Disabling preserves programs and user-owned skills, removing only Choruz-managed links from isolated profiles on their next launch. This is not an execution sandbox or an OS permission revocation.

On Unix hosts, Agent launches append the execution user's existing $HOME/.local/bin to PATH, preserving inherited precedence. prepare_harness_account_env links the device-installed browser-skill and cua-driver into isolated Claude/Codex profiles before execution; discovery and login probes remain read-only. Sources are searched in .agents/skills, .claude/skills, then .codex/skills under that device's home. Existing profile entries take precedence. The runtime shares neither credentials nor Harness settings, downloads nothing and grants no permissions. Default accounts and other Harnesses use their normal installed skill directories. Install the tools and skills separately on each execution device; a headless server without a supported browser or desktop does not acquire those capabilities through pairing. Restart an already-running Harness to discover newly installed skills. See BrowserSkill and Cua for host installation and consent requirements.

Terminal bridges write terminal.attached and terminal.detached audit rows with one attachment ID, binding/device/account context, duration, transferred byte counts and successful resize count. The REST input endpoint records terminal.submit_started and terminal.submit_finished with one submission ID. Success means input was written, not that the model completed a turn. Raw input, output and reconstructed prompts are not recorded. Output counts include replay. The bridge stops both forwarding tasks before recording final counters. A storage failure logs terminal activity persistence failed without changing the terminal operation's outcome; crashes can leave unmatched start records.

Route Handler Notes
GET/POST /v1/runtime/bindings, GET /v1/runtime/bindings/{binding_id}, POST .../rebind handlers_runtime Humans only; create_runtime_binding returns the existing active binding instead of creating a second; validates device and account. Binding views and bootstrap share list_binding_views; Claude/Codex use session, other terminal drivers use terminal, and explicit message bindings retain their message interface.
GET/POST /v1/runtime/bindings/{binding_id}/session, POST .../session/commands handlers_sessions Authorized direct members open/read a structured session or send an instance-fenced command on the binding's device.
PUT /v1/runtime/bindings/{binding_id}/host handlers_runtime_hosts::assign_binding_host remote-control plugin
GET/PUT /v1/runtime/policies/{conversation_id} handlers_runtime::get_runtime_policy / upsert_runtime_policy auto_mode, max_auto_turns, max_workflow_turns, require_human_after_n_turns, allow_agent_to_agent, allow_file_write, default_reviewer_agent_id, default_coordinator_agent_id, untagged_human_mode
GET /v1/conversations/{conversation_id}/runtime-status handlers_runtime_status Per-agent busy/queued/idle
POST /v1/companies/{company_id}/reset-sessions handlers_runtime::reset_company_sessions Clears external_session_id on the company's bindings
GET /v1/ws/terminals/{binding_id}?token=&cols=&rows= handlers_terminals::websocket_terminal Bridges browser bytes to the PTY on the binding's device, local or linked; POST /v1/terminals/{binding_id}/ensure and POST .../input reuse the same pool
GET /v1/ws/runtime-hosts/link host_link::websocket_host_link A paired device's persistent link; the first frame is a hello with the host token
PUT/DELETE /v1/conversations/{conversation_id}/hide handlers_conversations::hide_agent_session / restore_hidden_agent_session Per-user conversation_hidden row; agents are forbidden
POST /v1/runtime-hosts/{host_id}/harness-accounts, POST .../harness-accounts/{account_id}/verify handlers_runtime_hosts::register_harness_account, verify_harness_account Called by the runtime host
POST /v1/companies/{company_id}/harness-accounts/{account_id}/probe handlers_harness_logins::probe_harness_account Reads identity, models and exact quota on the device that holds the login (HostRequest::HarnessProbe, choruz_harness_login::probe_account) and stores the snapshot; a failure is recorded on the account and answered 409
POST /v1/companies/{company_id}/harness-accounts/{account_id}/logins, GET .../{login_id}, POST .../{login_id}/callback, POST .../{login_id}/cancel handlers_harness_logins Browser side of a sign-in, on this device or a runtime host; cancel closes an open login so a new one can start before the 15-minute expiry
POST /v1/runtime-hosts/{host_id}/harness-account-logins/claim, .../{login_id}/publish, .../{login_id}/callback/claim, .../{login_id}/complete, .../{login_id}/fail handlers_harness_logins Connector side of a sign-in on a runtime host
POST /v1/workspace-sessions/scan, POST /v1/workspace-sessions/import handlers_workspace_sessions Scan native session stores, import selected sessions as terminal DMs
GET /api/drivers/availability, GET /api/drivers/models, /api/harness-accounts[...] Next.js routes Web-side driver and account surface

Terminal path: authorize_terminal_binding requires a human caller, a terminal driver, a non-disabled binding, a direct conversation containing both caller and agent, an enabled agent and matching accessible workspaces; RuntimeHost::ensure_terminal hands a TerminalSpec to the binding's device, where choruz_host_runtime::ensure_terminal evicts dead children, reuses a live TerminalSession, or spawns the binary with terminal_cli_args (Codex adds --config check_for_update_on_startup=false and a managed CODEX_HOME), TERM=xterm-256color, CHORUZ_SEND, DISABLE_AUTOUPDATER, PI_SKIP_VERSION_CHECK and the harness account env. Codex sessions are captured by begin_codex_terminal_capture and anchored by write_terminal_session_anchor so a later ensure_terminal resumes the same native session.

Structured path: Claude/Codex use AgentSessionView for native text, tool details and pending interactions. handlers_sessions reuses terminal authorization and sends SessionRequest through RuntimeHost; the execution device owns the process and its journal. See the package contract for persistence and bounds, and the design decision for direct/group isolation.

Headless path: the pipeline's spawn_headless_session resolves the active binding and its headless external_session_id. Claude forks a verified direct session when seeding its first headless turn; it never writes that turn into the direct native transcript. Both local and remote completion fence late identity writes against the binding's captured configuration. Web provisioning (apps/web/lib/agents/agent-provisioning.ts, POST /api/agents/provision) creates the principal, workspace, helper, instruction file and binding.

Invariants

Headless binding progress follows active session leases across the agent's conversations. Local and remote execution project running; final batch completion, retry backoff and lease expiry project idle. Paused, disabled and error bindings are not overwritten. Bootstrap and binding sync updates expose the same stored state. in_flight_turn_id does not represent a multi-turn batch. See the progress ownership decision.

Invariant Pinned by
One non-disabled binding per agent (agent_runtime_bindings_one_per_agent partial unique index); a second create returns the existing one binding_defaults_and_uniqueness_are_enforced, binding_creation_waits_for_disable_and_rejects_the_disabled_agent in crates/choruz-application/tests/runtime_store.rs
Disabling an agent's bindings is atomic and idempotent disabling_agent_bindings_is_atomic_and_idempotent
workspace_path is normalised and guarded; state transitions follow can_transition_to workspace_paths_are_normalized_and_guarded, state_transitions_are_guarded
Administrative binding state changes and rebinds write audit_log entries; automatic headless progress uses the sync feed state_changes_and_rebind_write_audit_entries, headless_progress_tracks_leases_in_binding_snapshots_and_sync
A terminal session anchor is accepted only for the binding generation that captured it and only for the same workspace terminal_session_anchor_preserves_unrelated_config_and_validates_binding, terminal_session_anchor_rejects_delayed_capture_after_reset_touch, codex_terminal_anchor_rejects_same_native_session_for_same_workspace_binding
Terminal routes never write conversation_events; terminal bytes bypass the pipeline terminal_routes_do_not_write_conversation_events in services/choruz-api-gateway/src/tests/runtime.rs
A binding may reference only an active harness account of the same company, driver and host, and only a model listed in models_json; the trigger stamps harness_account_name and harness_account_profile_kind validate_runtime_binding_harness_account (V035); harness_account_binding_trigger_rejects_unverified_models
At most one open login per account; OAuth tokens never enter the database harness_account_login_open_account_idx; table comments in V035 and V036
Native import is idempotent per (workspace_path, driver_type, native_session_id) and serialised by an advisory lock key native_session_import_runs_end_to_end_and_is_idempotent, native_session_import_lock_key_executes_against_postgres
Imported sessions are terminal bindings with an explicit binary_path V032, V033
Hiding a session is a per-user view preference that emits a sync_change and never stops the agent trg_conversation_hidden_sync_change (V034); api_restore_hidden_conversation helper in tests.rs
PTY sessions persist until the child exits; there is no idle eviction evict_stale_terminals doc comment in crates/choruz-host-runtime/src/terminal.rs
A policy's default_coordinator_agent_id must be a valid principal policy_rejects_invalid_default_coordinator, policy_defaults_and_upsert_work

Failure modes

Background experience learning

Claude Code and Codex structured chats expose Experience learning. The owner selects another analysis Agent and enables background review. Review uses that Agent's account and quota; source content is sent to its model provider. The selected Agent's live conversation is not resumed.

The gateway worker reads native transcript windows on the target device and feedback from conversations shared by the owner and target Agent. It preserves a cursor and rolling summary, then content-reviews proposed guidance. Reviewed guidance applies only to later Choruz structured or headless submissions. Raw PTY keystrokes and existing project instruction files are not modified. History distinguishes content review from evidence about how a previously active revision performed.

No-change reports can carry an open objective across windows without citing evidence. Missing historical citations are checked against original records before the committed source cursor, using the same native session, account and workspace or currently shared feedback conversations. This also recovers citations omitted from earlier reports when the original source remains available. Summary text alone is never evidence. Unknown objective openers, unknown failure citations and missing current failure evidence have separate diagnostics; a new problem observation always requires failure evidence in the current window.

A revision-use marker from an earlier native window is re-read even when its identifier was retained. Validation requires a user record containing the active revision's marker before the current window, and cited failure evidence in the current window of the same session. Historical records remain separate from new evidence and are not added to the analysis prompt. A source reset cannot establish this cross-window ordering. Recovery exceeding the source reader's response limit fails without committing partial evidence.

For an observed mistake, an isolated search session receives only short problem categories and finds published prompt techniques. The source-reading reviewer adapts applicable findings or devises guidance when the completed search has no relevant result. A failed search preserves the prior revision and retries; it does not count as an empty search. Distinct episode references and native revision-use markers distinguish recurrence after an intervention from repeated reports about the original failure.

Content review records whether a distinct recurrence after guidance was used qualifies for structural search; it does not create collaborators. Future structured and headless turns capture the selected revision and run its team on the target device and account before the main Agent. These calls add latency and quota use; their proposals are not proof that checks passed. Structured Stop cancels preparation without sending the task. Prompt and team selection roll back together. Collaborators do not re-execute historical tasks, alter project files or grant permissions.

Disable learning to stop future application. Clear active revision removes the current selection without deleting history or disabling future analysis; Restore this revision selects a reviewed historical revision. Each action fences out an analysis result started under older settings. A disconnected device, unavailable native transcript or analysis failure preserves the previous revision and appears in the panel; Refresh history reads the latest state without discarding unsaved settings.

Learning checks emit learning.check audit records with a shared trace_id, per-call identifiers, stage, duration and success or a closed failure category. Unstructured device error text is not copied into these records. Revision validation carries that identifier, review_details (the reviewer report and the individual acceptance predicates), and escalation_decisions (the observed episode, intervention revision, applied revision and eligibility reason). A missing completion denotes interruption or unavailable persistence, not success. Input fingerprints and source references identify content held by the existing source owners; audit records do not copy raw transcripts, reconstructed prompts or account configuration. Rejected reviews stay candidates, and failures before revision creation remain in the audit store.

The contract is implemented by the worker, fixed learning procedures, device-local execution, execution roles and the scoped store. The learning decision, diagnostic decision and execution-role decision explain isolation and validation limits.

External native consumers can use the CLI preparation command or POST /v1/runtime/bindings/{bindingId}/experience/prepare. It reads the same active pointer and uses the same device-local collaborator owner, returning labeled input for a later native invocation. Only the learning owner can prepare that binding. The response is not a submitted turn, an execution receipt or a recurrence observation. Selection changes during preparation reject its result; already dispatched calls cannot be recalled. Byte-level terminal input remains untouched.

Behavior community

The learning settings can opt into reuse for new tasks in the same project, device and account. Matching tasks reference the existing learning owner: source rollback reaches them, and their authorized traces and owner feedback join its background analysis. New-task creation does not copy learning state or browser permissions. Context changes stop sharing. Resetting a native session keeps its captured source available for later curation while the original file and access remain valid.

Measured learning needs explicit task and analysis models. The same settings panel selects them on existing Agents, rather than requiring replacement tasks. Selection refuses active turns, closes idle native sessions and updates the selected models atomically. Captured session identity and history remain; the conversation view reconnects with the new model.

Task reuse does not expand an existing public-contribution permission. Turning it on clears that permission and invalidates outstanding undispatched publication work. The community panel discloses shared task history and permits fresh consent; already dispatched publication outcomes are not automatically retried.

The learning panel organizes confirmed problems into local cards with context, expected behavior, observed failure, execution-model attribution and versioned solutions. Three independent permissions control accepted-community search, automatic solution trials and public contribution. Local learning does not require community access. The typed exchange contract is shared by local and public records; private source references stay outside it. Missing native model metadata remains absent rather than inheriting the configured model.

Background preparation re-reads cited native records and owner feedback before generating a card. Adaptation searches local experience before accepted community records cached from immutable Hugging Face revisions. Candidate mappings receive fixed content review, and adapted instructions use the existing revision, evaluation and rollback path. Version-pinned community team suggestions enter the optimizer only after its existing recurrence gate allows team changes. Nothing installs arbitrary scripts or expands account permissions.

Encounter, application, effectiveness, ineffectiveness and recurrence are separate evidence kinds. Counts deduplicate independent objective and solution identities; later negative evidence suppresses an earlier effective count for that objective. Application outcomes require a native revision-use marker and a separate evidence review. The latest 200 displayed local records are not a population-wide model failure rate. A reviewed or saved solution alone is not proof of successful use.

Public contribution creates a separate redacted projection, checks its identity and recognized credentials, and runs an independent privacy review before dispatch. Redaction is fallible; only enable contribution for work authorized for public sharing. Contributions to ai-bad-behavior-library remain pending until accepted records appear in the dataset. Network failures preserve the accepted local cache. Retry preparation and review retries blocked, undispatched work; an uncertain dispatched request is not automatically resubmitted. See the operator setup and storage ownership.

Fixed task evaluation

The learning owner can submit a reviewed candidate and a task suite to POST /v1/runtime/bindings/{bindingId}/experience/evaluations. GET on the same path returns comparison history. The wire contract defines case limits, exact-text, JSON-equality and AI-judge checks. A judge check contains a reference result and task-specific acceptance criteria. The separate tool-free judge uses the fixed judge skill, without candidate guidance. It returns a reason and pass, fail or inconclusive verdict. Inconclusive judgments terminate the run with judge_inconclusive, retain the explanation and do not produce a numeric candidate score. Training, validation and test inputs are distinct; expected outputs stay in the evaluator, not the task prompt. Each comparison freezes both guidance snapshots and the suite before returning 202.

The background worker alternates baseline and candidate on each case using the target's explicit model and account through RuntimeHost. Model calls use empty scratch conversations with native tools disabled. Captured team members run before the final executor through the same runner as foreground work. Self-contained single-step and multi-step tasks need no external environment. Model output and assessments are retained in owner-scoped evaluation history; account configuration is not. A judge's opinion is not a formal proof.

An explicitly supplied environment adds a bounded command/observation loop in a disposable Docker container on the selected device. The environment fixes an already-installed image ID, UTF-8 starting files, step budget and verification commands. The container has no network, credentials or live-workspace mount; the starting files are mounted read-only and copied into its bounded writable workspace. Model actions inspect or modify this copy. Fixed checks run after the final answer and must all succeed, in addition to the output assessment. Reports retain commands, observations and check outcomes. Missing images, Docker failures or exhausted budgets fail the run; execution never falls back to the live host. Historical extraction does not invent missing environments. Docker is an explicit dependency of the replay integration test.

Changing learning settings or the active revision cancels pending results. A changed model, account selection, driver, binary path, workspace or device fails the next case. Provider-side model updates or external changes inside an account profile are not detected by this configuration fingerprint. A lost execution lease marks the comparison failed, retaining completed cases without automatically charging another call. The owner can submit a new comparison after resolving a failure. The evaluation decision explains the isolation boundary.

Reflective optimization

The evaluation request accepts an optional optimization configuration. It freezes the policy's analyst and runs a bounded candidate search in the same durable worker. Training minibatches supply visible rollout feedback to a tool-free proposer; full validation scores retain nondominated candidates and select the final winner. Optional merge rounds combine complementary parents. Test cases run only after selection and cannot change the winner. A proposal edits the executor instruction or the internal team, never the analyst or evaluator.

Evolve the internal execution team consents to structural search after reviewed recurrence, or for an existing reviewed team. The owner bounds total agents, including the final executor. Team candidates contain individual prompts and serial or parallel ordering. Serial members see prior findings; parallel members are independent. Members use the target model, account and device without task tools. The selected team runs before future structured DM and headless group tasks, with a 70-second total preparation deadline. Stop or member failure prevents foreground submission. History displays member prompts and ordering; prompt and team roll back together. See the team decision.

Each action reserves budget in its checkpoint before dispatch. Metric calls count task evaluations; model_calls_reserved includes collaborators, the maximum replay steps, independent judging, proposals and final review, including uncertain calls. A replay that finishes early can use fewer calls than reserved. Fresh comparisons are the default; cache_evaluations explicitly reuses candidate/case results. Queued checkpoints continue after restart; an interrupted in-flight call fails without automatic replay. The detailed evaluation report retains candidate ancestry, observations, budgets and winner. A manually requested search collects evidence without application. See the search decision.

The learning panel's Evaluate and optimize against a fixed suite option saves tasks and budgets. Automatically build tasks from work episodes uses the existing analyst's objective decomposition instead of manual answers. Each case keeps its opening reference, evidence and answer rationale. A separate tool-free review checks answer support, pre-answer context and privacy. Unsupported objectives carry a null check; later corrections replace or withdraw the same objective rather than creating a retry as a new problem. Enabling trace mode rereads retained history. The bounded corpus uses up to 64 objectives from the latest 100 case-bearing reports, within 48 KiB; missing source evidence is not reconstructed from a summary.

The fixed analyst classifies task type, capability, step structure and outcome separately. Incomplete work is not an error attribution. Reviewed equivalent-problem links, normalized identical inputs and shared evidence form groups; each group contributes at most one task. Different answers or an answer withdrawal within a group exclude that group until resolved. Group partitions prefer training over validation over test when connected evidence merges. Category round-robin sampling preserves minority task types and multi-step work within each split's budget, rather than selecting only frequent or easy tasks.

Enabled trace learning starts a retained-source review cycle daily, after the preceding paged sweep completes. Cursor checkpoints resume across restart. A completed sweep withdraws answers that were not revalidated; missing original evidence is not inferred from summaries. Reports retain the bounded corpus snapshot, content version, category/outcome counts, additions, updates, withdrawals and conflict counts in validation.dataset. Learning history displays these summaries. Versions describe data contents, while report timestamps record reviews of unchanged data. Original reports are retained; the bounded working set is not an exhaustive archive.

Task admission uses a blind, tool-free trial followed by the fixed task-quality-review.md procedure on the selected analysis Agent's device and account. The trial sees inputs and identifiers, not reference answers or source records. Review decides each task separately; a failed trial alone does not exclude a supported difficult task. A proposed input/check repair receives another blind trial and independent review before admission. One repair round is attempted per sweep; unresolved tasks keep their evidence but receive a null check and cannot score. validation.task_quality retains the procedure digest, reviewed inputs, trial outputs and each decision. These extra calls run under the existing background lease, which lasts fifteen minutes; no new scheduler or foreground session is used.

The analyst may propose one same-objective paraphrase per task. Only training groups receive a separate blind trial and fixed source/meaning review of that variant; repairs are disabled for variants. Rejection preserves the original task. A retained variant replaces the training input without adding a vote or changing its reference; validation and test use originals. Variant reviews are stored beneath validation.task_quality.variants. The extra variant batch costs at most two model calls per sweep.

Training sampling balances category and empirical difficulty. Counts use up to 100 completed evaluation runs with the current executor fingerprint and active guidance, the exact retained task snapshot, and baseline candidate only. Each run contributes at most one scored answer per objective; changed inputs, errors, inconclusive answers and held-out results do not count. Below three samples the band is insufficient; at least 80% correct is easy, at most 20% is hard, otherwise mixed. These bands describe observed performance, not intrinsic task difficulty or independent statistical certainty. The learning panel shows counts and scope; without an explicit model no performance report is available. Difficulty never changes holdout membership or the frozen queued suite.

Runs wait until all three splits have admissible groups, and freeze the selected cases and provenance within the evaluation record. The suite name includes the corpus version, checked again under the policy lock before automatic queueing. Later changes to used cases or new reviewed links to them cancel pending evaluations; completed reports retain their original snapshot. Scores are historical regression evidence, not performance on genuinely unseen tasks: the analyst may already have seen the source work. Automatically extracted cases are self-contained; replay requires explicit starting-state artifacts.

Background analysis stores its reviewed candidate instead of applying it immediately; an unchanged instruction can seed trace-based search without inventing a mistake. The evaluation worker queues it once per policy generation. Automatically apply a measured and reviewed improvement separately consents to application. A winner must strictly improve mean validation score and not reduce mean held-out score, then pass fixed content review of its guidance and team. Held-out results only gate application. They never select another winner or enter reflection. Without consent, evaluation remains evidence-only.

Application creates a revision with its seed's source references and evaluation provenance, records the first applied problem intervention, and changes the existing active pointer atomically. It invalidates older analysis claims. Clear and restore remain the rollback controls. History exposes call reservations, split scores and application decisions; a passing small suite is not a general capability claim. Per-run budgets do not cap lifetime or token usage. See the application decision.

Runtime failures

Failure Behaviour Signal
CLI binary not installed terminal spawn fails; headless turn dead-letters with driver_unavailable; GET /api/drivers/availability reports unavailable with setupHint availability endpoint, executor_command_failed
Gateway restart PTY pool is lost; the next ensure_terminal respawns and reconciles Codex capture metadata PTY child process exited, recreating session warn; codex_terminal_open_reconciles_capture_metadata_after_gateway_restart_window
Stale external_session_id headless resume-failure phrases clear the id; sync_session_id_from_disk (startup backfill_session_ids) and the terminal path's HostRequest::LatestSession plus record_discovered_session_id repair from the device's native session store; reset_company_sessions clears a whole company not resuming CLI session warn
Binding workspace missing on disk headless turn fails without retry binding workspace_path does not exist on disk
Harness authentication fails status becomes reauth_required or error; a sign-in can end failed, cancelled or expired before a verified identity is stored account list in the web UI
Model or quota refresh fails after authentication the login remains verified and the account remains active; the last complete snapshot and probed_at are preserved gateway or connector warning; account refresh action in the web UI
Terminal opened by a non-member, an agent, a group conversation or a disabled binding 403 from authorize_terminal_binding HTTP status
Binding on a runtime host whose connector holds no link 409 runtime host … is not connected from RuntimeHost::for_binding; the browser retries when the device reconnects HTTP status, host link established log
Lease lost during a headless turn handled by the pipeline lease monitor (see message-pipeline) lease expired warn

Tests

Related