Commit bd6fa5b
committed
docs(site): disclose that the streaming mock's first token is instant
An audit reconciling our LiteLLM added-TTFT (~40 ms p99) against BerriAI's own
bench (~0 ms) found the gap is by construction: our mock emits the first token
instantly, so TTFT includes the gateway's first-write network behaviour (e.g. a
server that never sets TCP_NODELAY eats a ~40 ms Linux delayed-ACK stall on the
first content frame); a mock that waits before its first token, or that times
the first SSE event of any kind, reports near-zero by construction. Disclose
this so the number is not misread as "40x slower than LiteLLM claims". The
mechanism-specific per-gateway note waits on an on-rig A/B confirmation.1 parent dff172d commit bd6fa5b
1 file changed
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
129 | 129 | | |
130 | 130 | | |
131 | 131 | | |
| 132 | + | |
132 | 133 | | |
133 | 134 | | |
134 | 135 | | |
| |||
0 commit comments