feat(evals): a NanoClaw image, the controller's failure reason, and evaluations on GKE - #980
Merged
Conversation
… require `@moltzap/evals` requires MOLTZAP_NANOCLAW_IMAGE as a digest-pinned reference for half the matrix, and nothing produced one: the NanoClaw container contract in `agents/nanoclaw/runtime.ts` named an entrypoint, a bootstrap config, a state directory, and a gateway port that no artifact implemented. `scripts/build-nanoclaw-image.mjs` stages the pinned upstream checkout, the workspace MoltZap channel and its two packed workspace dependencies, the eval provisioner, and the skill, then prints the same digest-pinned identity the controller producer prints. The entrypoint materializes a writable project root — NanoClaw resolves `data/`, `groups/`, and its own manifest from the cwd — seeds the eval agent group, starts NanoClaw, and republishes its owner-local CLI socket on the fixed bridge port. The relay carries bytes, not a second protocol: the CLI channel already speaks the NDJSON frames the controller's gateway decodes. A built image was run end to end: the provisioner applied NanoClaw's seventeen migrations and initialized `eval-agent`, NanoClaw reached `NanoClaw running`, `data/cli.sock` appeared, and the bridge accepted a connection on 18790. NanoClaw runs every agent turn in a container it spawns itself and refuses to start when no runtime answers, so a cell also needs a reachable container runtime; the image carries the client and the platform supplies the endpoint. Both image failures in the evaluation CLI now name the script that produces the reference, because neither is a value an operator already has. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Failure detail was sanitized three times and then discarded. `cluster/watch.ts` collects up to 200 bounded, credential-redacted lines of controller output; a decodable failure summary makes that a result rather than an activity failure, so the detail was dropped where it was collected, the submitter's final JSON line had no field for it, and the evaluation CLI wrote a canned sentence into SQLite. The run's namespace is deleted on the way out, so nobody could go read the Job's log afterwards either — an operator saw only "the simulator controller reported an infrastructure failure". The reason now rides the two attempts that already exist to record one. `RunFailedAttempt.detail` and `LedgerAllocationFailedAttempt.failure.detail` carried a constant sentence; they carry the controller's own account when there is one, and `phoenix-run.ts` was already publishing exactly those two fields as a run's error, so the reason reaches Phoenix without a line of new plumbing. Three shapes collapse on the way: `ControllerObservation` had one `failed` member with an optional result and a detail that was dead whenever the result was present — two states wearing one tag, free to disagree. It is now `completed` (a decodable result, successful or not) and `failed` (no result, and this is the only account of why). The composed failure detail is sanitized once over the whole string rather than per part. Bounding the pieces separately let their concatenation exceed the bound, and the trim then ate the front — which is where the Job's condition reason is. The submitter's bound trims the byte array directly and steps off continuation bytes. A UTF-16 slice cannot express a byte count, and the previous convergence loop re-encoded the string up to fourteen times to work around that. `isEntryModule` — which both simulator profile entrypoints already use — now guards the evaluation CLI, so `cli.ts` can be imported. The image-producer vocabulary moves back beside the environment it describes, and the attempt constructors are testable at all for the first time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ul README Running an evaluation sweep against GKE was an undocumented sequence an operator had to reconstruct: publish the controller image, resolve its pushed digest, open a port-forward that survives being dropped, and export six variables — five of which are properties of the cluster they had just attached to. Getting one wrong measures a different cluster than the one they think. `cluster.sh evals` performs that sequence and hands the rest of its arguments to the sweep, reusing the image publish and the forward supervisor `run` already owns. `cluster.sh publish-image` exposes the publish alone, printing only the digest so a caller can assign it. The NanoClaw image stays a passthrough: it is the runtime under evaluation, not this cluster's infrastructure. Not `exec`: the forward supervisor is a background job of this shell, and replacing the shell would strand it with no trap left to reap it. Two README defects go with it. It claimed the profile has no Temporal deployment while `setup` applies one, and it claimed live GKE qualification could not be proven after a hundred-agent run had completed. The replacement says what the decision log actually supports — the run happened, its ledger is retained nowhere a reader can check — and leaves the gate open on the evaluation half rather than on all of it. `MOLTZAP_TEMPORAL_CLUSTER_ADDRESS` was documented nowhere, and it is the one endpoint no other file explains: the worker Pod cannot reach a localhost forward, and a worker that never connects looks exactly like an empty queue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mage intact
An MCP server became either a stdio spawn or a streamable-HTTP endpoint while
this series was in flight, and the container runtime now passes the definition
through untouched. The image entrypoint still rebuilt the stdio shape by hand,
so a `{name, url}` server reached NanoClaw as a command that does not exist and
a URL that was never written down.
It is not this file's definition to rebuild. Rekeying by name and forwarding the
rest verbatim also means the pinned NanoClaw revision — which understands the
stdio shape only — starts honouring a URL server on a revision bump, with no
second change here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d payload `run` and `evals` each carried their own copy of the prelude that reaches the cluster — attach, publish, forward, trap. The trap is the part that reaps a leaked port-forward, and a forward that leaks still accepts connections while proxying to a pod that no longer exists, so two copies meant a fix could reach one verb and miss the other. One `begin_cluster_session` owns it. The image build had two spellings of one rule: the repository argument was checked for `@` when it arrived and for `@` or whitespace when the reference was built, so a whitespace repository passed the first and failed the last statement after a 45-minute build. `assertRepository` is now the only rule, and the test that grepped the old spelling out of the source text calls it instead. The entrypoint materializes its project root concurrently — every entry targets a distinct path, and this runs before the bridge port opens, which is what the controller reads as readiness. `data` leaves the mutable list, because the copied install already carries it with its stamped marker. The provisioner uses spawn's own timeout rather than a hand-rolled timer, and `startNanoClaw` stops returning a child handle no caller signals. The final image drops ~32MB no agent turn reads: upstream's marketing imagery, its translated docs, and the SQLite C amalgamation better-sqlite3 keeps beside the prebuild it loads. The agent pool autoscales from zero, so that was pulled onto a fresh node for every run. Verified by rebuilding and running the image: gateway listening on 18790, upgrade marker stamped, workspace skill seeded, and both MCP server shapes intact in NanoClaw's container config. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three gaps between the evaluation matrix and something that can run it.
The NanoClaw application image nothing built
packages/evalsrequiresMOLTZAP_NANOCLAW_IMAGEas a digest-pinned referencefor sixteen of thirty-two cells, and no artifact satisfied it. The container
contract in
agents/nanoclaw/runtime.tsnamed an entrypoint, a bootstrapconfig, a state directory, and a gateway port that nothing implemented.
scripts/build-nanoclaw-image.mjsstages the pinned upstream checkout, theworkspace MoltZap channel with its two packed workspace dependencies, the eval
provisioner, and the skill, then prints the digest-pinned identity the
controller producer already prints. The entrypoint materializes a writable
project root — NanoClaw resolves
data/,groups/, and its own manifest fromthe cwd — seeds the eval agent group, starts NanoClaw, and republishes its
owner-local CLI socket on the bridge port. The relay carries bytes, not a
second protocol: the CLI channel already speaks the NDJSON frames the
controller's gateway decodes.
Built and run end to end: the provisioner applied NanoClaw's seventeen
migrations and initialized
eval-agent, NanoClaw reachedNanoClaw running,data/cli.sockappeared, the bridge accepted a connection on 18790, and bothMCP server shapes reached NanoClaw's container config intact.
A NanoClaw cell also needs a reachable container runtime. NanoClaw executes
every agent turn in a container it spawns itself and refuses to start when no
runtime answers. The image carries the client and reads
DOCKER_HOST; nothingin
cluster/kubernetes/objects.tsgives a Sandbox a daemon today, so thatremains open and is the one thing between "the image starts" and "sixteen
NanoClaw cells run".
The failure reason that was collected and then discarded
cluster/watch.tscollects up to 200 bounded, credential-redacted lines ofcontroller output. A decodable failure summary makes that a result rather than
an activity failure, so the detail was dropped where it was collected, the
submitter's stdout line had no field for it, and the evaluation CLI wrote a
canned sentence into SQLite. The run's namespace is deleted on the way out, so
the Pod log was gone too — an operator saw only "the simulator controller
reported an infrastructure failure".
The reason now rides the two attempts that exist to record one.
RunFailedAttempt.detailandLedgerAllocationFailedAttempt.failure.detailheld a constant; they hold the controller's own account when there is one, and
phoenix-run.tsalready published exactly those two fields as a run's error, soit reaches Phoenix with no new plumbing. No new error class, no new attempt
kind, no migration.
Three shapes collapse on the way.
ControllerObservationhad onefailedmember with an optional result and a detail that was dead whenever the result
was present — two states wearing one tag, free to disagree; it is now
completedandfailed. The composed failure detail is sanitized once over thewhole string rather than per part, because bounding the pieces separately let
their concatenation exceed the bound and the trim then ate the Job's condition
reason. The submitter's bound trims the byte array and steps off continuation
bytes, since a UTF-16 slice cannot express a byte count.
isEntryModule— which both simulator profile entrypoints already use — guardsthe evaluation CLI, so
cli.tscan be imported and its attempt constructors aretestable at all for the first time.
Evaluations on GKE, and a README that was wrong twice
Running a sweep against GKE was an undocumented sequence: publish the controller
image, resolve its pushed digest, hold a port-forward that survives being
dropped, export six variables — five of which are properties of the cluster the
operator had just attached to.
cluster.sh evalsperforms it and hands the restof its arguments to the sweep;
cluster.sh publish-imageexposes the publishalone. The NanoClaw image stays a passthrough: it is the runtime under
evaluation, not this cluster's infrastructure.
Not
exec: the forward supervisor is a background job of that shell, andreplacing the shell would strand it with no trap left to reap it.
The README claimed the profile has no Temporal deployment while
setupappliesone, and claimed live GKE qualification could not be proven after a
hundred-agent run had completed. The replacement says what the decision log
actually supports — the run happened, its ledger is retained nowhere a reader
can check — and leaves the gate open on the evaluation half rather than all of
it.
MOLTZAP_TEMPORAL_CLUSTER_ADDRESSwas documented nowhere and is the oneendpoint no other file explains: a worker Pod cannot reach a localhost forward,
and a worker that never connects looks exactly like an empty queue.
Verification
build,lint(incl. knip),format:check,typecheck,arch:check,docs:check:drift,docs:check,docs:check:mermaid,docs:check:gates-test,test:compute-next-version,simulator-packages.mjs, all four profile/imagechecks, and
nx affected -t typecheck:tests test test:integration(simulator45 files/280 tests, evals 13/102).
🤖 Generated with Claude Code