Skip to content

test(live): treat error as a terminal and skip models this account cannot use - #981

Open
kaizhou-lab wants to merge 1 commit into
mainfrom
test/set-model-survives-entitlement-gaps
Open

test(live): treat error as a terminal and skip models this account cannot use#981
kaizhou-lab wants to merge 1 commit into
mainfrom
test/set-model-survives-entitlement-gaps

Conversation

@kaizhou-lab

Copy link
Copy Markdown
Contributor

run_backend_set_model waited only for a finish frame, so a turn that ended in an error burned the full 300s and then reported "the turn did not finish" — a message naming the wrong problem. On 2026-09-07 that very nearly went into the record as a codex 0.153.4 regression.

What is actually happening

Probing the frames settled it. The turn does terminate, and the reason does reach the user:

frames: ["start", "tips", "tips", "tips", "tips", "tips", "error"]

tips:   "Reconnecting... n/5 — unexpected status 403 Forbidden: user not allowed to
         access model. This user can only access models=['standard'].
         Tried to access gpt-6-astra"

error:  {"code":"USER_LLM_PROVIDER_PERMISSION_DENIED",
         "ownership":"user_llm_provider",
         "detail":"unexpected status 403 Forbidden: user not allowed to access model…"}

There is no product defect. An earlier note suspecting the app-server swallowed the 403 was wrong and is retracted — the backend already classifies this as the user's provider's problem and says so, with the model name and the allowed set.

Two changes, both to the test

1. error is a terminal. The wait ends on either terminal, and the failure message carries the backend's own detail plus the warnings that preceded it. Failure time on this account: 314s → 28s, and the message now names the entitlement rather than a phantom hang.

2. Try every candidate model, not just the catalog's first. The catalog lists models an account may not be entitled to, and its order is not stable: 0.153.4 led with gpt-6-astra (403 here), 0.151.0 led with gpt-5.6-sol, and two runs tonight disagreed with each other. Picking the first made the test's subject vary with the release — so an A/B across versions was not comparing the same thing at all, which is precisely how the false regression arose.

The part that had to be discovered by running it

A turn ending in error leaves the conversation with no active agent, so the next switch 404s:

{"success":false,"error":"No active agent for this conversation","code":"NOT_FOUND"}

The retry therefore re-ensures the runtime first, through the same endpoint the test already uses to start it. This was not assumed — the first iteration attempt failed exactly there, and an earlier attempt at this fix (2026-09-07) was withdrawn unshipped for the same reason before the cause was understood.

Verification

run result
codex 0.153.4, single test ok, 26.71s — gpt-6-astra refused and recorded, gpt-5.6-sol completed
set_model_takes_effect, all backends 2/2 ok (claude + codex; agy has no such test)

The helper is shared across backends, so both were run rather than only the one being fixed.

Clippy clean, fmt clean.

Relation to the codex bump

This is one of the two things keeping codex's gate B from being green. The other is #976 (the plan tool became opt-in). Neither depends on the other; both are needed before VERIFIED_CODEX_VERSION can move off 0.151.0.

Test-only change — left for human review, no auto-merge.

…nnot use

run_backend_set_model waited only for a `finish` frame, so a turn that ended in
an `error` burned the full 300s and then reported "the turn did not finish" --
a message naming the wrong problem. On codex 0.153.4 that is exactly what
happened, and on 2026-09-07 it was very nearly filed as a release regression.

Probing the frames settled it. The turn does terminate, and the reason does
reach the user:

  frames: ["start", "tips", "tips", "tips", "tips", "tips", "error"]
  tips:   "Reconnecting... n/5 - unexpected status 403 Forbidden: user not
           allowed to access model. This user can only access models=['standard'].
           Tried to access gpt-6-astra"
  error:  code USER_LLM_PROVIDER_PERMISSION_DENIED,
          ownership user_llm_provider

So there is no product defect -- an earlier note suspecting the app-server
swallowed the 403 was wrong, and is retracted. The backend already classifies it
as the user's provider's problem and says so.

Two changes, both to the test:

1. `error` is a terminal. The wait now ends on either terminal and the failure
   message carries the backend's own detail plus the warnings that preceded it.
   Failure time on this account: 314s -> 28s.

2. Try every candidate model, not just the catalog's first. The catalog lists
   models an account may not be entitled to, and its ORDER is not stable --
   0.153.4 led with gpt-6-astra (403 here) while 0.151.0 led with gpt-5.6-sol,
   and two runs tonight disagreed with each other. Picking the first made the
   test's subject vary with the release, so an A/B across versions was not
   comparing the same thing at all.

A turn that ends in `error` leaves the conversation with no active agent, so the
next switch would 404 with NOT_FOUND. The retry re-ensures the runtime first,
through the same endpoint the test already uses to start it. That was found by
running it, not assumed: the first iteration attempt failed exactly there.

Verified: codex 0.153.4 26.71s ok (gpt-6-astra refused, gpt-5.6-sol completed);
claude and codex together 2/2 ok. The helper is shared, so both were run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant