Judge a stream's speed on what it generates visibly (#133)

Fixes to how a stream's speed is judged against its provider's minimum:

- Counts the upstream reports are no longer used. Gemini's running
  output count includes the thoughts it does not show; once it leaped
  ahead of the text, the text counted for nothing and a fast stream
  could be cut.
- Summarised reasoning is left out: Responses reasoning summaries,
  Anthropic thinking blocks, Gemini thought parts and Bedrock reasoning
  content arrive in bursts while the model reasons at full speed. No
  second is judged while such a phase is open, and its time and text are
  left out of the window after. Reasoning streamed as generated (Chat
  reasoning_content, Responses reasoning_text) still counts.
- A long pause is a quarter of the window, at most 5 s, and no second is
  judged whose last content is more than a second old: the burst an
  upstream sends after holding tokens back is waited for.
- The answer's end is also seen by the format's last frame (Anthropic
  message_stop, Chat [DONE], Responses response.completed), not only by
  a stop reason.
- The upstream's connection is dropped at the cut, before the error
  frame goes to the client.
- The affinity is dropped and the speed written to the log row only when
  the stream's outcome is the cut.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Author: fylorn

CHANGELOG.md

@@ -98,19 +98,25 @@ endpoints of its own under `/api/admin/security/redact/exceptions`.
   `min_speed_tokens_per_sec` 1–1000 and `min_speed_window_secs` 5–120,
   default 20, on the provider endpoints; blank or `null` is off, the
   default). From the first token on, a stream is judged once a second:
-  when the text it streamed over the last window (answer, reasoning and
-  tool arguments, four ASCII characters or one other character to a
-  token; Gemini's running count when higher) comes to fewer tokens per
-  second than the minimum, the stream ends with a 503 error in the
+  when the visible text it streamed over the last window (answer,
+  reasoning streamed as generated, and tool arguments; four ASCII
+  characters or one other character to a token) comes to fewer tokens
+  per second than the minimum, the stream ends with a 503 error in the
   client's format (`overloaded_error` for Anthropic, `response.failed`
   for Responses, `UNAVAILABLE` for Gemini), the upstream connection is
-  closed, and the route is charged a failure. Pauses of 5 s or more are
-  left out of the window, and a window that is more than half pauses is
-  not judged. The session's affinity to that route is dropped, the
-  answer so far is billed as for any stream cut short, and the log row
-  carries the error `UpstreamTooSlow` and the speed at the cut
-  (`tokens_per_sec`). Whole answers and Responses WebSocket turns are
-  not measured.
+  closed, and the route is charged a failure. Left out of the window:
+  pauses of a quarter of the window or more (at most 5 s), and
+  summarised reasoning (Responses reasoning summaries, Anthropic thinking
+  blocks, Gemini thought parts, Bedrock reasoning content), which arrives
+  in bursts while the model reasons; a window more than half left out is
+  not judged. Nor is a second while summarised reasoning is open, one
+  whose last content is more than a second old, or anything after the
+  upstream has ended the answer. Counts the upstream reports are not
+  used: Gemini's includes its hidden thoughts. The session's affinity to
+  that route is dropped, the answer so far is billed as for any stream
+  cut short, and the log row carries the error `UpstreamTooSlow` and the
+  speed at the cut (`tokens_per_sec`). Whole answers and Responses
+  WebSocket turns are not measured.
 
 ### Changed
 

crates/gateway/src/lifecycle/mod.rs

@@ -426,8 +426,10 @@ pub(crate) fn build_chat_pump(
                         }) else {
                             continue;
                         };
-                        // Too slow: the client is told in its own format, and
-                        // returning drops the upstream's connection.
+                        // Too slow: the upstream's connection goes now, not once
+                        // the client has taken the error, which is said in its
+                        // own format.
+                        drop(source);
                         let err = too_slow(&provider, rate, min);
                         let mut out = shaper.process(&error_frame(client, err.status_code(), &err.to_string()));
                         out.extend(shaper.finish());