Skip to content

Commit 463a88d

Browse files
committed
fix(sim): align S7/S10 with the 64fad04 task cap and verify contract
S7 assumed the old static 3000-char cap; file transports are uncapped unless EXOMA_MAX_TASK_CHARS_NO_ARGV sets one, so the flow now sets prompt_builder's cap for that call only. S10 wrote no verify/tests.json, so the runtime kept the verdict incomplete; it now writes an allowlisted green test and checks the runtime outcome. sim_flows: 33 of 33 on Windows. DEC-083 records the result.
1 parent a7cc5b6 commit 463a88d

2 files changed

Lines changed: 44 additions & 5 deletions

File tree

‎docs/research/decisions/DEC-083-exoma-1-0-0-stable-staged-at-64fad04.yaml‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,15 @@ evidence_notes: |
6363
refused before delegation) and S10 (delegated verify pass) mismatch. The same two mismatch
6464
against 4ed5a2b in an isolated environment, so they are not introduced by 64fad04; their cause
6565
was not investigated here.
66+
Result added 2026-11-06: both were stale sim expectations, not runtime defects. S7 assumed the
67+
old static 3000-char cap; the opencode file transport is uncapped unless
68+
`EXOMA_MAX_TASK_CHARS_NO_ARGV` sets a cap (`exoma/core/prompt/prompt_builder.py` `_transport_cap`,
69+
`_no_ceiling`). S10 wrote no `verify/tests.json`, so the runtime added a `tests:` gap and the
70+
verdict stayed incomplete (`exoma/core/evidence/contract.py`, verdict rule: pass only with no
71+
warnings). The fixture in `tools/sim/sim_flows.py` now sets the cap to 3000 for S7 only and
72+
writes an allowlisted green test for S10; `python -m tools.sim.sim_flows` against 64fad04 on
73+
Windows gives 33 of 33 as expected (S7 kept 3000/12018 and refused; S10 verdict pass, runtime
74+
test passed, exit 0; S10b still fail, exit 2).
6675
6776
implementation:
6877
version: exoma 64fad04 (1.0.0)
@@ -98,7 +107,7 @@ trade_offs:
98107
limitations:
99108
- One delegated run, one provider and model, on Windows only; no Linux or macOS staging here.
100109
- Real use on 64fad04 is outside this record; the release gate stays open until it runs.
101-
- sim_flows S7 and S10 mismatch on both pins; until explained, that harness is not evidence for those two flows.
110+
- sim_flows S7 and S10 mismatch on both pins; until explained, that harness is not evidence for those two flows. (Explained 2026-11-06 as stale expectations and fixed in the fixture; see evidence_notes.)
102111
revisit_when:
103112
- real use on 64fad04 shows a regression from de217f8 or 67727e7
104113
- sim_flows S7 or S10 is traced to a runtime defect rather than a stale expectation

‎tools/sim/sim_flows.py‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
extract_session_id = OpenCodeAdapter.extract_session_id
4545
from exoma import check # noqa: E402
4646
from exoma import cli as main # noqa: E402
47+
from exoma.core.prompt import prompt_builder # noqa: E402
4748

4849
ROWS = []
4950

@@ -417,8 +418,17 @@ def simulate():
417418
)
418419

419420
# --- S7 task truncation ----------------------------------------------
421+
# File and stdin transports (opencode, codex) carry no task cap by default; a cap
422+
# applies only when EXOMA_MAX_TASK_CHARS_NO_ARGV sets one. That value is read at
423+
# import and bound into prompt_builder, so the flow sets the module global for this
424+
# call: it checks the refusal path under a policy cap, not the uncapped default.
420425
huge = "analyse this dump " + ("x" * 12000)
421-
truncated = main.run("analyze", huge, "sim-session", work_dir)
426+
saved_cap = prompt_builder.DEFAULT_MAX_TASK_CHARS_NO_ARGV
427+
prompt_builder.DEFAULT_MAX_TASK_CHARS_NO_ARGV = 3000
428+
try:
429+
truncated = main.run("analyze", huge, "sim-session", work_dir)
430+
finally:
431+
prompt_builder.DEFAULT_MAX_TASK_CHARS_NO_ARGV = saved_cap
422432
meta = truncated.get("meta", {})
423433
record(
424434
"S7",
@@ -459,20 +469,40 @@ def simulate():
459469
)
460470

461471
# --- S10 verify delegated: pass and fail --------------------------------
472+
# A pass needs a test request the runtime ran green: without verify/tests.json the
473+
# runtime records a `tests:` gap and the verdict stays incomplete.
462474
config["commands"]["verify_mode"] = "delegated"
475+
config["commands"]["verify_test_commands"] = [f'"{sys.executable}"']
463476
write_config(project, config)
477+
tests_request = (
478+
workflow_paths(project, "verify-pass-session")["session_dir"] / "verify" / "tests.json"
479+
)
480+
tests_request.parent.mkdir(parents=True, exist_ok=True)
481+
tests_request.write_text(
482+
json.dumps(
483+
{
484+
"commands": [f'"{sys.executable}" -c "pass"'],
485+
"reason": "sim: one allowlisted command that exits 0",
486+
}
487+
),
488+
encoding="utf-8",
489+
)
464490
adapter.body_for["verify"] = VERIFY_PASS
465491
adapter.default_body = VERIFY_PASS
466492
vpass = main.run(
467493
"verify", "confirm the change landed", "verify-pass-session", work_dir
468494
)
495+
vmeta = vpass.get("meta", {})
496+
test_outcomes = (vmeta.get("runtime_tests") or {}).get("outcomes") or []
497+
test_status = test_outcomes[0].get("status") if test_outcomes else None
469498
record(
470499
"S10",
471500
"delegated verify pass",
472-
"meta.verdict=pass, exit code 0",
473-
f"verdict={vpass.get('meta', {}).get('verdict')}, "
501+
"meta.verdict=pass, runtime test passed, exit code 0",
502+
f"verdict={vmeta.get('verdict')}, test={test_status}, "
474503
f"exit={main._verify_exit_code('verify', vpass, 'verify')}",
475-
vpass.get("meta", {}).get("verdict") == "pass"
504+
vmeta.get("verdict") == "pass"
505+
and test_status == "passed"
476506
and main._verify_exit_code("verify", vpass, "verify") == 0,
477507
)
478508

0 commit comments

Comments
 (0)