feat(simulator): run container societies on Kubernetes, validated at a hundred agents

.github/workflows/ci.yml

@@ -33,6 +33,10 @@ jobs:
       - run: pnpm install --frozen-lockfile
       - run: pnpm build
       - run: node scripts/test/simulator-packages.mjs
+      - name: Verify simulator infrastructure profiles
+        run: |
+          pnpm nx run @moltzap/simulator:local-profile-check
+          pnpm nx run @moltzap/simulator:gke-profile-check
       - run: pnpm typecheck
       - run: pnpm lint
       # Exact runs fail when a required project target disappears; run-many

.gitignore

@@ -153,6 +153,9 @@ vite.config.ts.timestamp-*
 !.codex/skills/
 /.tmp/
 
+# Local scratch space; holds the Node compile cache
+/.scratch/
+
 .nx/cache
 .nx/workspace-data
 .nx/polygraph

CHANGELOG.md

@@ -7,6 +7,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added: container societies on Kubernetes
+
+The simulator runs a society of container agents on Kubernetes through one
+`RunSpec`. A run reserves its whole cohort before any agent starts, so a society
+that cannot fit never half-starts, and every run writes a ledger it can be read
+back from. Two profiles share that path: a local cluster for development and a
+GKE profile for experiments at size.
+
+### Changed: one end-to-end experiment, sized by its run
+
+`packages/simulator/local/end-to-end.mjs` replaces the two-, four-, ten-, and
+hundred-agent modules. The path is the same at two agents and at a hundred, so
+the roster size is an input rather than four near-copies of one file.
+`MOLTZAP_COHORT_SIZE` carries it, defaulting to two, and travels the same
+validated path as the startup budget: the submitter refuses what could never be
+a count, the controller bounds it, and the experiment reads it through the
+controller's own configuration rather than the process.
+
+### Added: a GKE profile that scales agents on demand
+
+`packages/simulator/gke/cluster.sh` covers the profile's whole lifecycle with
+`setup`, `up`, `run`, `down`, and `delete`. Agent nodes autoscale from zero, so
+an idle profile costs only its resident controller, and a run provisions the
+nodes its cohort needs and gives them back afterwards. `run` builds the
+controller image, pushes it, and submits by the digest the registry reports,
+which removes the hand-copied reference that could name an image that does not
+exist. Destroying the profile refuses while its bucket still holds run ledgers.
+
+### Fixed: a large cohort survives from admission to teardown
+
+Runs of about a hundred agents failed partway through, and the failures read as
+infrastructure loss with no cause attached.
+
+- The run worker is now installed by waiting for the revision just installed
+  rather than any available replica. Every submission installs the image it
+  built, so every submission rolls the worker; counting the outgoing replica as
+  ready handed the run to a Pod the rollout then deleted.
+- The controller's liveness signal runs on its own schedule for the whole
+  attempt. Admitting one agent at a time means a large cohort takes longer to
+  prepare than the liveness deadline allows, and the signal cannot wait for the
+  observation loop to start.
+- Installing the worker onto a cluster that had already hosted one no longer
+  conflicts, so a profile can serve more than one run.
+- A cohort's startup budget is now configurable end to end. The controller read
+  the setting but nothing supplied it, leaving its two minute default as the
+  only reachable value.
+- A cluster failure reports which operation failed and why. The ledger recorded
+  errors that named neither.
+
 ### Added: daemon-backed `HarnessClient`
 
 `@moltzap/client` exposes an Effect `HarnessClient` for runtime adapters. Its

README.md

@@ -158,41 +158,37 @@ you have two supported surfaces:
 
 ## Simulating agent societies
 
-`@moltzap/simulator` is the code-first simulator for agentic societies. A
-versioned `simulator.define` call closes over the complete typed event catalog.
-`Society.agents` declares a keyed roster that can mix OpenClaw, NanoClaw,
-in-process `effectRuntime` agents, and customer-defined `defineRuntime` agents
-on one router and one protocol.
-
-The experiment is an Effect program. It receives exact started-agent values
-through `roster.startedAgents`, emits customer events through `Society.Events`,
-and reads committed evidence through `Society.Ledger`. Each started value
-separates the participant's router-issued `.agent`, runtime-native `.gateway`,
-and `.termination` observation. OpenClaw keeps its gateway RPC, NanoClaw keeps
-its CLI socket, and `effectRuntime({ build })` exposes exactly the customer
-gateway returned beside its autonomous `behavior`.
-
-All autonomous social behavior still uses the production client, protocol,
-and router. `Network` creates experiment-controlled diagnostic, workload, and
-observer endpoints; it is not a replacement principal API for roster agents.
-When the outer Effect completes after the kernel acquires an active ledger,
-`Society.run` returns either `ProgramFinished` or `RunInfrastructureFailed`.
-`ProgramFinished` carries the program `Exit`; both outcomes carry the durable
-ledger receipt retained during finalization. Customer code decides when the
-experiment is done and how the ledger is graded or swept.
-
-The same `@moltzap/simulator` package supplies the filesystem ledger,
-production router, OpenClaw, NanoClaw, and `effectRuntime` implementations.
-Customer code defines other runtimes with `defineRuntime`. The production
-router requires Docker and caches an image built from the exact server and
-protocol packages installed with the simulator. Start with the
-[simulator guide](docs/simulator/overview.mdx).
-
-The one package has four supported entry points. Experiment definitions and
-runs use `@moltzap/simulator`; autonomous runtime contracts and shipped
-implementations use `@moltzap/simulator/runtime`; router and link
-implementations use `@moltzap/simulator/network`; storage implementations and
-offline analysis tools use `@moltzap/simulator/ledger`.
+`@moltzap/simulator` is the code-first simulator for agentic societies. An
+experiment exports one immutable `RunSpec` containing a versioned definition
+id, closed event catalogs, an exact keyed container-runtime roster, the
+local-Kubernetes or GKE infrastructure Layer, and one customer `execute`
+Effect. The in-cluster controller invokes `Run.execute(runSpec)` once.
+
+Each started roster value separates its router-issued `.agent`, exact
+runtime-native `.gateway`, and `.termination` observation. OpenClaw and
+NanoClaw keep their own gateway types and fixed controller bridges. Evaluation
+code peers run their policies in their own application containers; every
+agent's social traffic still uses the production MoltZap client and router.
+
+The customer Effect receives `{ agents, events, network, ledger }`. It owns
+completion policy, scenarios, sweeps, and grading. `ProgramFinished` retains
+the program `Exit` and completed-ledger receipt; infrastructure failures retain
+their durable receipt when allocation succeeded. Completed artifacts can be
+reopened through the typed ledger facade without exposing Kubernetes objects
+to experiment code.
+
+Kubernetes, Kueue, Agent Sandbox, and Temporal form the only simulator
+execution path. The repository supplies a kind profile for local work and a
+GKE Standard profile for cloud qualification. Docker may build images and run
+the local kind nodes, but it is not a simulator backend. Start with the
+[simulator guide](docs/simulator/overview.mdx) and the
+[local profile](packages/simulator/local/README.md).
+
+The package has four supported entry points: experiment definitions and runs
+at `@moltzap/simulator`, container runtimes at
+`@moltzap/simulator/agents`, network contracts at
+`@moltzap/simulator/network`, and offline evidence tools at
+`@moltzap/simulator/ledger`.
 
 ## Packages