test(live): treat error as a terminal and skip models this account cannot use - #981
Open
kaizhou-lab wants to merge 1 commit into
Open
test(live): treat error as a terminal and skip models this account cannot use#981kaizhou-lab wants to merge 1 commit into
kaizhou-lab wants to merge 1 commit into
Conversation
…nnot use
run_backend_set_model waited only for a `finish` frame, so a turn that ended in
an `error` burned the full 300s and then reported "the turn did not finish" --
a message naming the wrong problem. On codex 0.153.4 that is exactly what
happened, and on 2026-09-07 it was very nearly filed as a release regression.
Probing the frames settled it. The turn does terminate, and the reason does
reach the user:
frames: ["start", "tips", "tips", "tips", "tips", "tips", "error"]
tips: "Reconnecting... n/5 - unexpected status 403 Forbidden: user not
allowed to access model. This user can only access models=['standard'].
Tried to access gpt-6-astra"
error: code USER_LLM_PROVIDER_PERMISSION_DENIED,
ownership user_llm_provider
So there is no product defect -- an earlier note suspecting the app-server
swallowed the 403 was wrong, and is retracted. The backend already classifies it
as the user's provider's problem and says so.
Two changes, both to the test:
1. `error` is a terminal. The wait now ends on either terminal and the failure
message carries the backend's own detail plus the warnings that preceded it.
Failure time on this account: 314s -> 28s.
2. Try every candidate model, not just the catalog's first. The catalog lists
models an account may not be entitled to, and its ORDER is not stable --
0.153.4 led with gpt-6-astra (403 here) while 0.151.0 led with gpt-5.6-sol,
and two runs tonight disagreed with each other. Picking the first made the
test's subject vary with the release, so an A/B across versions was not
comparing the same thing at all.
A turn that ends in `error` leaves the conversation with no active agent, so the
next switch would 404 with NOT_FOUND. The retry re-ensures the runtime first,
through the same endpoint the test already uses to start it. That was found by
running it, not assumed: the first iteration attempt failed exactly there.
Verified: codex 0.153.4 26.71s ok (gpt-6-astra refused, gpt-5.6-sol completed);
claude and codex together 2/2 ok. The helper is shared, so both were run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
run_backend_set_modelwaited only for afinishframe, so a turn that ended in anerrorburned the full 300s and then reported "the turn did not finish" — a message naming the wrong problem. On 2026-09-07 that very nearly went into the record as a codex 0.153.4 regression.What is actually happening
Probing the frames settled it. The turn does terminate, and the reason does reach the user:
There is no product defect. An earlier note suspecting the app-server swallowed the 403 was wrong and is retracted — the backend already classifies this as the user's provider's problem and says so, with the model name and the allowed set.
Two changes, both to the test
1.
erroris a terminal. The wait ends on either terminal, and the failure message carries the backend's owndetailplus the warnings that preceded it. Failure time on this account: 314s → 28s, and the message now names the entitlement rather than a phantom hang.2. Try every candidate model, not just the catalog's first. The catalog lists models an account may not be entitled to, and its order is not stable: 0.153.4 led with
gpt-6-astra(403 here), 0.151.0 led withgpt-5.6-sol, and two runs tonight disagreed with each other. Picking the first made the test's subject vary with the release — so an A/B across versions was not comparing the same thing at all, which is precisely how the false regression arose.The part that had to be discovered by running it
A turn ending in
errorleaves the conversation with no active agent, so the next switch 404s:The retry therefore re-ensures the runtime first, through the same endpoint the test already uses to start it. This was not assumed — the first iteration attempt failed exactly there, and an earlier attempt at this fix (2026-09-07) was withdrawn unshipped for the same reason before the cause was understood.
Verification
gpt-6-astrarefused and recorded,gpt-5.6-solcompletedset_model_takes_effect, all backendsThe helper is shared across backends, so both were run rather than only the one being fixed.
Clippy clean, fmt clean.
Relation to the codex bump
This is one of the two things keeping codex's gate B from being green. The other is #976 (the plan tool became opt-in). Neither depends on the other; both are needed before
VERIFIED_CODEX_VERSIONcan move off 0.151.0.Test-only change — left for human review, no auto-merge.