fix: end a round when the provider is down instead of reviewing with it - #117
Merged
Conversation
An OAuth token was revoked mid-round on 2026-08-13. `correctness` had already approved; the next nine rubrics each recorded the CLI's own "Failed to authenticate. API Error: 401 OAuth access token has been revoked." as their result, five seconds apart, and the round rendered a scoreboard with nine blocking `error` rows and posted it to the PR. Across the stores this had happened to 639 rubric runs and 107 whole rounds, and the commonest text was not a status code at all but "You've hit your session limit · resets 9:30pm (UTC)". Two gaps. error_kind read only stderr, but Claude Code in -p mode puts a total provider failure in the result text with an empty stderr, so every one of those classified as `unknown_error`; it now also reads a result short enough to be the whole diagnosis, under the same bound TauCetiWorker uses, and knows the subscription CLIs' prose for an exhausted plan. And nothing stopped the round: two consecutive rubrics failing for a reason that will not change now abort it before anything is rendered or posted, with a distinct exit status the CLI reports as itself. The error case files stay: they are the provenance, and they are what re-runs those rubrics next round. What changes is that an outage no longer reaches a PR. Replayed over all 6141 stored rounds the breaker fires on 85, skipping 424 model calls, and never fires on a round without an errored rubric. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T1zpkPwUh41oU3mtSQGDVU
Three ways the breaker could fire on something that was not an outage, or fail
to fire on one.
Length was doing the work of a trust boundary. `text` is model output over an
untrusted diff, so a short review that never emitted its marker ("the /login
handler returns 401 on an expired token") read as an auth failure. Require a
structured failure signal first: claude's own is_error, codex's parsed error
status, or a non-zero exit. Every observed failure carries one; a model talking
about a 401 while its CLI reports success no longer does.
The streak was global while providers are chosen per rubric, so a claude auth
failure plus a codex rate limit declared the whole service down. Track it per
provider and abort only once every configured one is confirmed unavailable,
which is what the abort tells its caller.
And it counted rubrics, so a reply or contest round dispatching a single rubric
could never trip it and still rendered the error scoreboard this exists to
prevent. Count attempts: run_rubric already retries, so one rubric confirms the
diagnosis twice on its own.
Replayed over all 6155 stored rounds this still fires on 85 and skips 424 model
calls, with no round lacking an errored rubric touched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T1zpkPwUh41oU3mtSQGDVU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR ends a round when the provider is unusable, instead of filing the outage as a review.
An OAuth token was revoked mid-round on 2026-08-13.
correctnesshad already approved; the next nine rubrics each recorded the CLI's own "Failed to authenticate. API Error: 401 OAuth access token has been revoked." as their result, five seconds apart. The round then rendered a scoreboard headed "changes requested" with nineerrorrows and posted it, and an errored rubric blocks the merge, so an expired credential became a merge blocker on somebody else's PR. Across the stores this had happened to 639 rubric runs and 107 whole rounds, and the commonest text was not a status code at all but "You've hit your session limit · resets 9:30pm (UTC)".Two things were missing.
error_kindread only stderr, but Claude Code in-pmode puts a total provider failure in the result text with an empty stderr, so every one of those classified asunknown_error. It now also reads a result short enough to be the whole diagnosis, under the same bound and for the same reason as TauCetiWorker'sclassify_agent_failure, and it knows the subscription CLIs' prose for an exhausted plan. A long review that merely quotes a status number is unaffected.And nothing stopped the round. Two consecutive rubrics failing for a reason that will not change by calling again now abort it before anything is rendered or posted. Two rather than one, because a single 401 can be a token rotating under a long round and the retry inside
run_rubricalready covers that; two in a row is the credential or the plan. Onlynot_authenticated,quota_exhausted, andrate_limitedcount. Overload, timeouts, and transport failures are per-call and keep the existing behaviour.The error case files stay as written. They are the provenance of what happened, they are what makes the next round re-run those rubrics, and TauCetiReview#105 is the standing argument that a total auth failure must stay recorded and stay loud. What changes is that none of it reaches the PR. The abort exits with a status distinct from an ordinary engine failure so the CLI reports the cause rather than a generic command failure, and its last log line names the cause in the terms a driving worker classifies on.
Replayed over all 6141 rounds in the local stores, the breaker fires on 85 and skips 424 model calls, and never fires on a round that had no errored rubric.
🤖 Prepared with Claude Code