You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bcc8637
Browse filesBrowse the repository at this point in the historyBrowse files
fix(charts): regenerate from the complete 14-gateway board
The final publish logged "matplotlib not importable in the charts venv - charts will NOT be
regenerated this run", so every PNG on the board was still drawn from YESTERDAY's data: chart files
timestamped 19:27 beside a bundle generated at 05:31. The numbers were right and the pictures were
of a different run, which is the shape of error this project exists to prevent - a published surface
disagreeing with the artifact it claims to draw.
SAME ROOT CAUSE AS THE SSH KEY EARLIER TONIGHT. The chart venv lives under $TMPDIR
(bench-charts-venv), and macOS reaps that directory by age. It did not delete the venv outright - the
directory survived with a broken pip and a hollow matplotlib package, which is why the import failed
rather than the path being missing. That is the second thing $TMPDIR has cost this run, after the
private key that made the last gateway unreachable, and it is now the strongest argument for moving
both out of there.
Venv rebuilt from scratch (repairing in place failed: pip itself was gutted) and every chart redrawn
from the committed snapshots. check-consistency PASS, bench-audit PASS on 14 gateways, 204 site
tests, 0 failures.
|[LiteLLM · Rust](https://github.com/BerriAI/litellm)| 98 µs | 48,394 <sub>(+1% from 1 ms to no bound)</sub> | - | - |`litellm-ai-gateway`|
10
-
|[Busbar](https://github.com/GetBusbar/busbar)|114 µs |47,557 <sub>(+4% from 1 ms to no bound)</sub> | 7 MiB |241 MiB |`getbusbar/busbar:1.4.1`|
10
+
|[Busbar](https://github.com/GetBusbar/busbar)|115 µs |48,959 <sub>(+3% from 1 ms to no bound)</sub> | 7 MiB |248 MiB |`getbusbar/busbar:1.4.1`|
11
11
|[agentgateway](https://github.com/agentgateway/agentgateway)| 227 µs | 25,090 <sub>(+8% from 1 ms to no bound)</sub> | 25 MiB | 47 MiB |`ghcr.io/agentgateway/agentgateway:v1.4.0`|
12
12
|[AISIX (api7)](https://github.com/api7/aisix)| 267 µs | 17,746 <sub>(+0% from 1 ms to no bound)</sub> | 67 MiB | 358 MiB |`target/release/aisix`|
13
13
|[Helicone](https://github.com/Helicone/ai-gateway)| 284 µs | 15,770 <sub>(+7% from 1 ms to no bound)</sub> | 43 MiB | 56 MiB |`target/release/ai-gateway`|
@@ -31,7 +31,7 @@ The most req/s each gateway carried while 99% of requests finished under the col
31
31
| Gateway | p99 < 1 ms | p99 < 5 ms | p99 < 10 ms | p99 < 50 ms | p99 < 100 ms | no bound | at 10 ms: concurrency, observed tail |
@@ -54,7 +54,7 @@ Every rung of the same sweep the frontier readings above are taken from, summari
54
54
| Gateway | req/s at lowest c | peak req/s (at c) | gain (rate × / concurrency ×) | saturates (95% of peak) | p99 at lowest c → at top c | first c that failed a request | top c probed |
55
55
|---|--:|--:|--:|--:|--:|--:|--:|
56
56
|[LiteLLM · Rust](https://github.com/BerriAI/litellm)| 8,680 at c=1 | 48,351 at c=64 | 5.6× / 64× | c=16 | 131 µs → 4.86 s | none | c=32768 |
57
-
|[Busbar](https://github.com/GetBusbar/busbar)| 7,572 at c=1 |47,433 at c=64 | 6.3× / 64× | c=16 |148 µs → 0 µs|c=512| c=512|
57
+
|[Busbar](https://github.com/GetBusbar/busbar)| 7,909 at c=1 |48,700 at c=64 | 6.2× / 64× | c=16 |144 µs → 4.44 s|none| c=32768|
58
58
|[agentgateway](https://github.com/agentgateway/agentgateway)| 4,484 at c=1 | 24,993 at c=64 | 5.6× / 64× | c=16 | 279 µs → 595 ms | none | c=32768 |
59
59
|[AISIX (api7)](https://github.com/api7/aisix)| 4,293 at c=1 | 17,638 at c=16 | 4.1× / 16× | c=8 | 257 µs → 3.86 s | none | c=32768 |
60
60
|[Helicone](https://github.com/Helicone/ai-gateway)| 3,443 at c=1 | 15,710 at c=64 | 4.6× / 64× | c=16 | 316 µs → 32.9 ms | c=64 | c=256 |
@@ -77,7 +77,7 @@ Same box, same mock, one gateway at a time. Streaming figures are the overhead t
@@ -93,35 +93,35 @@ Same box, same mock, one gateway at a time. Streaming figures are the overhead t
93
93
94
94
**✕** cells are measured refusals, not gaps: the gateway was offered the load and could not do the thing (buffered instead of streaming, rejected the Anthropic shape, or has no native key/limit governance). **n/a** = that suite hasn't been run for this gateway yet.
Method: added latency = gateway p99 − direct-to-mock p99 at concurrency 1; a frontier reading = the highest req/s any probed concurrency carried while 99% of requests finished under the STATED bound and the gateway failed none it accepted (readings are published at 1, 5, 10, 50, 100 ms and with no bound; the columns above use 10 ms, and every caption names the bound it used); cost figures divide that 10 ms reading by $0.1632/hr for the pinned 4-core (m7g.xlarge) slice; RSS idle = after first 200, steady state = the level the RSS settled at under load. Same box, same mock, same load, one gateway at a time. Each gateway's source ref is pinned in its own `gateways/<name>/definition.json`; the built commit is in each row.
126
126
127
-
<sub>Page + charts regenerated **2026-07-31 05:28 UTC** from the raw `results/*.json`.</sub>
127
+
<sub>Page + charts regenerated **2026-07-31 05:39 UTC** from the raw `results/*.json`.</sub>
|[LiteLLM · Rust](https://github.com/BerriAI/litellm)| 98 µs | 48,394 <sub>(+1% from 1 ms to no bound)</sub> | - | - |`litellm-ai-gateway`|
10
-
|[Busbar](https://github.com/GetBusbar/busbar)|114 µs |47,557 <sub>(+4% from 1 ms to no bound)</sub> | 7 MiB |241 MiB |`getbusbar/busbar:1.4.1`|
10
+
|[Busbar](https://github.com/GetBusbar/busbar)|115 µs |48,959 <sub>(+3% from 1 ms to no bound)</sub> | 7 MiB |248 MiB |`getbusbar/busbar:1.4.1`|
11
11
|[agentgateway](https://github.com/agentgateway/agentgateway)| 227 µs | 25,090 <sub>(+8% from 1 ms to no bound)</sub> | 25 MiB | 47 MiB |`ghcr.io/agentgateway/agentgateway:v1.4.0`|
12
12
|[AISIX (api7)](https://github.com/api7/aisix)| 267 µs | 17,746 <sub>(+0% from 1 ms to no bound)</sub> | 67 MiB | 358 MiB |`target/release/aisix`|
13
13
|[Helicone](https://github.com/Helicone/ai-gateway)| 284 µs | 15,770 <sub>(+7% from 1 ms to no bound)</sub> | 43 MiB | 56 MiB |`target/release/ai-gateway`|
@@ -20,7 +20,7 @@ The most req/s each gateway carried while 99% of requests finished under the col
20
20
| Gateway | p99 < 1 ms | p99 < 5 ms | p99 < 10 ms | p99 < 50 ms | p99 < 100 ms | no bound | at 10 ms: concurrency, observed tail |
@@ -34,7 +34,7 @@ Every rung of the same sweep the frontier readings above are taken from, summari
34
34
| Gateway | req/s at lowest c | peak req/s (at c) | gain (rate × / concurrency ×) | saturates (95% of peak) | p99 at lowest c → at top c | first c that failed a request | top c probed |
35
35
|---|--:|--:|--:|--:|--:|--:|--:|
36
36
|[LiteLLM · Rust](https://github.com/BerriAI/litellm)| 8,680 at c=1 | 48,351 at c=64 | 5.6× / 64× | c=16 | 131 µs → 4.86 s | none | c=32768 |
37
-
|[Busbar](https://github.com/GetBusbar/busbar)| 7,572 at c=1 |47,433 at c=64 | 6.3× / 64× | c=16 |148 µs → 0 µs|c=512| c=512|
37
+
|[Busbar](https://github.com/GetBusbar/busbar)| 7,909 at c=1 |48,700 at c=64 | 6.2× / 64× | c=16 |144 µs → 4.44 s|none| c=32768|
38
38
|[agentgateway](https://github.com/agentgateway/agentgateway)| 4,484 at c=1 | 24,993 at c=64 | 5.6× / 64× | c=16 | 279 µs → 595 ms | none | c=32768 |
39
39
|[AISIX (api7)](https://github.com/api7/aisix)| 4,293 at c=1 | 17,638 at c=16 | 4.1× / 16× | c=8 | 257 µs → 3.86 s | none | c=32768 |
40
40
|[Helicone](https://github.com/Helicone/ai-gateway)| 3,443 at c=1 | 15,710 at c=64 | 4.6× / 64× | c=16 | 316 µs → 32.9 ms | c=64 | c=256 |
@@ -48,36 +48,36 @@ Same box, same mock, one gateway at a time. Streaming figures are the overhead t
**✕** cells are measured refusals, not gaps: the gateway was offered the load and could not do the thing (buffered instead of streaming, rejected the Anthropic shape, or has no native key/limit governance). **n/a** = that suite hasn't been run for this gateway yet.
Method: added latency = gateway p99 − direct-to-mock p99 at concurrency 1; a frontier reading = the highest req/s any probed concurrency carried while 99% of requests finished under the STATED bound and the gateway failed none it accepted (readings are published at 1, 5, 10, 50, 100 ms and with no bound; the columns above use 10 ms, and every caption names the bound it used); cost figures divide that 10 ms reading by $0.1632/hr for the pinned 4-core (m7g.xlarge) slice; RSS idle = after first 200, steady state = the level the RSS settled at under load. Same box, same mock, same load, one gateway at a time. Each gateway's source ref is pinned in its own `gateways/<name>/definition.json`; the built commit is in each row.
82
82
83
-
<sub>Page + charts regenerated **2026-07-31 05:28 UTC** from the raw `results/*.json`.</sub>
83
+
<sub>Page + charts regenerated **2026-07-31 05:39 UTC** from the raw `results/*.json`.</sub>
0 commit comments