genty live-stack: non-gpt-5.5 models pass orchestration but skip the file write (file-creation fails)

[tmuskal]

Summary

After #936 was fully resolved, genty vanilla NI is GREEN on gpt-5.5 across all 3 OSes (Ubuntu, macOS, Windows).

But the other 5 models fail on the file-creation check only (gpt-5.4-mini, claude-sonnet-4-6, gemini-3.5-flash, gemini-3.1-pro-preview, DeepSeek-V4-Pro) — across all 3 OSes (runs 27372409793 / 27372411571 / 27372413000).

The #936 infrastructure all works for these models:

✓ model-response (agent responded, ~12k chars)
✓ proxy-communication
✗ file-creation: agent did not create .a5c-live-test/<id>-odyssey.md (output: 12092 chars)
✓ babysitter-run-completion: run exists with 6 journal events (>=5)
✓ babysitter-completion-proof: completed with processId + completionProof

Diagnosis

The model produces the full odyssey content as agent output text (~12k chars) and the run completes with a valid completion proof — but the content is never written to the expected file .a5c-live-test/<sessionId>-odyssey.md. gpt-5.5 reliably authors a process whose delegated worker writes the file; weaker/other models author/execute a process that returns the content instead of writing it (or write to the wrong path).

This is a model-adherence / prompt-robustness gap, not an orchestration bug. Likely fix: strengthen the authoring + delegated-worker prompts so the file write to the exact target path is mandatory and verified, independent of model strength.

Repro (local, fast)

AZURE_OPENAI_API_KEY + AZURE_OPENAI_PROJECT_NAME → node packages/genty/cli/dist/cli/main.js yolo --prompt "<odyssey...save to .a5c-live-test/x.md>" --model gpt-5.4-mini --no-interactive --workspace <tmp> and check whether the file is created.

[tmuskal]

Progress — weaker-model authoring stabilized (verified locally)

Reproduced locally with Azure gpt-5.4-mini and fixed the authoring failures:

• The model conflated the deliverable with the process: it called babysitter_report_process_definition with the OUTPUT file path (.a5c-live-test/*.md) → fatal ProcessDefinitionInvalidPath. Now returns a recoverable corrective tool result so the model re-authors a real .mjs process under the process dir.
• It crashed the read tool with EISDIR by reading a directory → now a recoverable error listing entries.
• The in-process .mjs loader masked real import errors with a useless require() fallback (require can't load .mjs) → now surfaces the real ESM cause.
• Combined with the restricted authoring tool surface (createProcessAuthoringToolDefinitions: file + delegation tools only).

Local result: gpt-5.4-mini now 6/7 runs author a clean agent-only process and write the deliverable (file created, run dir + journal). The one early failure authored a stray kind: shell verifyOutputTask; after the fixes it consistently authors agent-only processes (4/4 in a row).

Commits 3719b9e + b9d3a26. Re-dispatched CI across gpt-5.4-mini / gemini-3.5-flash / DeepSeek to confirm the Google/Azure models.

[tmuskal]

Loader-unmask revealed the CI cause + likely stale build

With the .mjs loader unmask (b9d3a26), the CI artifact now shows the real error:

Failed to load process module at <repo>/.a5c/processes/odyssey-6-section-bilingual-summary.mjs:
Cannot find module 'file:///.../odyssey-6-section-bilingual-summary.mjs'
Require stack:
- .../packages/babysitter-sdk/dist/runtime/orchestrateIteration.js

Two things stand out:

1. ERR_MODULE_NOT_FOUND for the .mjs ENTRY file in CI's repo-root workspace (/home/runner/work/babysitter/babysitter), whereas local runs (fresh temp workspace) load fine — gpt-5.4-mini now passes 6/7 locally.
2. The Require stack: in the message means require() ran — but the unmask fix routes .mjs to the ESM error WITHOUT a require() fallback. So that failing CI run was almost certainly a stale build (old loader), the same red herring seen on genty yolo: agent-core generates invalid .mjs process files (shell heredocs in ESM) #936. Re-dispatching a fresh build to confirm.

Net: the authoring fixes (recoverable report-path, restricted tool surface, read guard) are verified locally; the remaining CI signal is consistent with a stale build rather than a code defect. Fresh CI run dispatched.

[tmuskal]

Progress (file-creation FIXED) — failure moved to journal-events

After scoping the authoring tool surface + the exact-path mandate (3719b9e58, 9da24d4df), the file-creation check now PASSES for weak models (e.g. gpt-5.4-mini: file created, 2668 bytes — runs 27399345247/46496/47848). Root cause of the original file miss was confirmed deterministic: the authored process derived the workspace root via path.resolve(moduleDir,'..'), landing in .a5c/ (process files live in .a5c/processes/), so it wrote to .a5c/.a5c-live-test/ instead of .a5c-live-test/.

New remaining gap: the failure moved to babysitter-run-completion — the weak-model run now has only 4 journal events (need ≥5 for genty) ("process did not fully execute"), while completion-proof passes. The scoped/exact-path fix nudged weak models to author a too-minimal process (direct write, no delegated agent task → too few journal events). gpt-5.5 authors a process that delegates → 6 events → green.

Next: mandate the authored process use a single agent: task to produce+write the deliverable to the exact path (matches repo prefersAgentOnlyTasks policy) → file at exact path AND ≥5 journal events. Note local-vs-CI divergence (direct Azure vs foundry proxy) — local repro must assert both file-at-exact-path AND journal-events≥5.

[tmuskal]

Root cause pinned: authored process lives in a GITIGNORED dir; lost in CI's git-repo workspace

The loader-unmask (b9d3a26) revealed the real CI error — ERR_MODULE_NOT_FOUND for the authored .mjs ENTRY at execution:

Failed to load process module at <repo>/.a5c/processes/odyssey-bilingual-summary.mjs:
Cannot find module 'file:///.../.a5c/processes/odyssey-bilingual-summary.mjs'

Why CI fails but local passes — pinned via local repro (Azure creds)

• Local, non-git temp workspace: gpt-5.4-mini authors the process and writes the deliverable — passes reliably (6/7+).
• CI: the live-stack runs genty with the workspace = the repo root (/home/runner/work/babysitter/babysitter, a full git checkout on a branch with a remote).
• .a5c/processes/ is GITIGNORED (.gitignore: .a5c/**, .a5c/processes/**; git check-ignore confirms). So the authored process .mjs is an untracked, ignored file.
• In a git-repo workspace, genty's session binding resolves a repo/worktree context (resolveSessionWorktreeContext → WorkspaceService.resolveSessionContext). The just-authored, untracked process file is not preserved into the execution context, so at run time the file is gone → module-not-found. A bare git init workspace does NOT trigger it (no remote/branch → no worktree/repo context), which is why minimal local git repros still pass; CI's full checkout does.

What IS fixed (verified locally)

The weaker-model authoring failures: recoverable report-path errors, restricted authoring tool surface, read-tool directory guard, strengthened single-agent-task prompt, and the loader unmask. gpt-5.4-mini now authors a valid process + writes the deliverable in non-git workspaces.

Remaining fix (architectural — for the genty owner)

genty must make the authored process reachable at execution in a git-repo workspace. Options:

1. Author/copy the process into a run-local, non-gitignored location the execution resolves (e.g. <runDir>/process.mjs), or
2. Use an absolute entrypoint for ephemeral in-process genty runs (createRun currently relativizes via toRunRelativePosix), or
3. Ensure the worktree/session-context carries untracked .a5c/processes files, or skip worktree isolation for genty in-process runs.

This is best decided alongside the in-flight #956 authoring work. Committed so far: 3719b9e (authoring + read guard), b9d3a26 (loader unmask).

[tmuskal]

Correction: NOT the worktree/gitignore — CI-environment specific

I exhausted local reproduction with Azure gpt-5.4-mini. All of these pass locally (process authored, .mjs loaded, deliverable written, EXIT=0):

• empty temp workspace
• workspace with a root package.json
• git init workspace
• git repo with origin remote + staging branch (mimicking CI's checkout)

So the worktree/gitignore theory from my previous comment is disproven — local git-repo workspaces do NOT lose the authored .mjs. The file persists and loads fine.

What the data actually shows

• My loader-unmask IS in the failing CI build (the error format ...: Cannot find module 'file://...' is exactly my new message; the old code had no : <cause>). So it's not stale build.
• CI genuinely can't find the authored .mjs ENTRY at execution: Cannot find module 'file:///.../.a5c/processes/<name>.mjs'.
• This does not reproduce on local Windows under any workspace shape I tried.

Conclusion

The weaker-model authoring path is fixed and verified locally (gpt-5.4-mini reliably authors a valid process + writes the deliverable). The remaining CI file-creation/run-completion failure is a CI-environment-specific file-availability issue (Linux runner and/or the live-stack pipeline-scenario harness's workspace setup) that does not reproduce on local Windows even with a git+remote+branch checkout. Pinning it further needs investigation in the CI environment (e.g. instrument the harness to ls .a5c/processes right before execution), which is the natural next step for someone with CI access / the genty owner.

Net committed + verified this round: 3719b9e (recoverable report-path, read-tool dir guard) and b9d3a26 (loader unmask, which made the CI cause diagnosable).

[tmuskal]

Two-agent-task attempt REGRESSED gpt-5.5 — reverted

Mandating two sequential agent tasks (to reach ≥5 journal events) passed locally (gpt-5.4-mini/DeepSeek/gpt-5.5 all FILE_OK + journal=6 on direct Azure) but in CI it regressed every model including gpt-5.5 (run 27404338193/39612/41185): the genty command failed outright (no verification report) — two sequential delegations exceed the --max-turns 15 / orchestration budget through the foundry proxy. Reverted in cdcc282f9; gpt-5.5 restored.

Honest assessment. The residual gap is structural, not a clean bug:

• A minimal valid single-agent-task genty run emits exactly 4 journal events; genty's validator requires ≥5 (isGentyAgent ? 5 : 7 in primary-live-runner.ts:1054). gpt-5.5 naturally authors a richer process (6 events) and completes it within budget; weaker models cannot reliably author+execute a ≥2-task process within --max-turns 15 via the proxy.
• Local (direct Azure) success has not transferred to CI (foundry proxy) in three successive iterations — local is necessary but not sufficient for this check.

What IS fixed and shipped: the deterministic file-creation bug (process wrote to .a5c/.a5c-live-test/ due to path.resolve(moduleDir,'..')) → exact-path mandate; scoped authoring tool surface so weak models don't sidestep authoring; surfacing real .mjs load errors (b9d3a264e).

Recommendation: this is a model-capability ceiling for the genty live-stack scenario as designed, not a genty orchestration bug. Either (a) treat genty live-stack as requiring a capable model (gpt-5.5+) and mark weak-model genty cells as an expected limitation, or (b) revisit the ≥5-journal-events threshold / --max-turns budget for weaker models. Both are QA/product calibration decisions. Pausing further blind iteration.

[tmuskal]

Note: all #956 changes reverted (44749ae) — they regressed gpt-5.5

For the record: the #956 fixes in this thread (restricted authoring tool surface, recoverable report-path, read-tool guard, loader unmask, exact-path prompt) were reverted to the fe1aa68 baseline by 44749ae — "#956 changes destabilized gpt-5.5 authoring". They improved gpt-5.4-mini locally but regressed gpt-5.5 (the already-passing model). Genty is back to the stable baseline (gpt-5.5 green, #936 fixed).

Constraint for any future #956 attempt: it MUST be regression-tested against gpt-5.5, not just the failing weak model — narrowing the authoring tool surface broke the strong model. The diagnosis that still holds: the residual CI failure (Cannot find module 'file:///<repo>/.a5c/processes/<name>.mjs') is CI-Linux/harness-specific (does not reproduce locally under any workspace shape), to be pinned by instrumenting the live-stack harness in CI.