You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a400fa0
Browse filesBrowse the repository at this point in the historyBrowse files
# Top 5 gateways (table: lowest added latency; each chart: top 5 by its own metric)
2
2
3
-
**Ran on:** AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway. · 2026-07-30T02:36:30Z
3
+
**Ran on:** AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway. · 2026-07-30T05:08:25Z
4
4
5
5
Every number below is regenerated from the raw `results/*.json` - re-run `run-all.sh` and this page updates. Passthrough and translation figures are the canonical per-gateway records (matrix per-cell sweep, perf/xlate-suite fallback) from `site/data.json`, the same values the site table ranks. Chart bars are **colored by implementation language** (Rust / Go / Python / Node / Other). **Rows are sorted by added latency (p99), lowest first.**
|[LiteLLM · Rust](https://github.com/BerriAI/litellm)| 106 µs | 44,363 <sub>(+1% from 1 ms to no bound)</sub> | - | - |`litellm-ai-gateway`|
10
-
|[agentgateway](https://github.com/agentgateway/agentgateway)| 216 µs | 24,298 <sub>(+7% from 1 ms to no bound)</sub> | 25 MiB | 47 MiB |`ghcr.io/agentgateway/agentgateway:v1.4.0`|
11
-
|[AISIX (api7)](https://github.com/api7/aisix)| 267 µs | 17,009 <sub>(+1% from 1 ms to no bound)</sub> | 67 MiB | 349 MiB |`target/release/aisix`|
12
-
|[Helicone](https://github.com/Helicone/ai-gateway)| 291 µs | 15,108 <sub>(+5% from 1 ms to no bound)</sub> | 43 MiB | 56 MiB |`target/release/ai-gateway`|
13
-
|[Kong](https://github.com/Kong/kong)| 396 µs | 22,659 <sub>(+79% from 1 ms to no bound)</sub> | 378 MiB | 591 MiB |`kong:3.9.3`|
**Throughput is a curve, not a number.** The column above is one reading of each gateway's concurrency sweep: the most req/s it carried while 99% of requests finished under **10 ms** and it failed **none** it accepted. The same sweep is published at 5 tail-latency bounds (1 ms, 5 ms, 10 ms, 50 ms, 100 ms) plus with no bound at all, and the shape across them is the comparison that matters: a gateway already at its ceiling at 1 ms is a different machine from one that doubles when given 5 ms. See the frontier-shape chart. **≥** on a number means the sweep's top rung won, so that rate is a floor and no ceiling was established.
12
+
**✕** = did not serve under load (0 successful req/s). **0** = came up, but no tested concurrency held p99 < 10 ms while failing none of the requests it accepted.
13
+
16
14
## The frontier: throughput at each tail you accept
17
15
18
16
The most req/s each gateway carried while 99% of requests finished under the column's bound **and it failed none it accepted**. Reading left to right is the tradeoff: a row that barely changes gives you its full rate at a tight tail, and a row that climbs steeply is buying throughput with latency. The last column applies no latency bound at all, so it answers only "how much before it starts failing requests". Rates are non-decreasing left to right by construction - relaxing a bound can only add qualifying rungs, never remove one.
19
17
20
18
| Gateway | p99 < 1 ms | p99 < 5 ms | p99 < 10 ms | p99 < 50 ms | p99 < 100 ms | no bound | at 10 ms: concurrency, observed tail |
**≥** = the sweep's top rung won, so that rate is a **floor** and no ceiling was established. **0** = the sweep ran and no rung held that bound while failing nothing. **n/a** = the record carries no reading at that bound. A **✕** cell names the record's own reason for the absence.
29
23
@@ -33,11 +27,7 @@ Every rung of the same sweep the frontier readings above are taken from, summari
33
27
34
28
| Gateway | req/s at lowest c | peak req/s (at c) | gain (rate × / concurrency ×) | saturates (95% of peak) | p99 at lowest c → at top c | first c that failed a request | top c probed |
35
29
|---|--:|--:|--:|--:|--:|--:|--:|
36
-
|[LiteLLM · Rust](https://github.com/BerriAI/litellm)| 8,121 at c=1 | 44,299 at c=32 | 5.5× / 32× | c=8 | 140 µs → 5.23 s | c=32768 | c=32768 |
37
-
|[agentgateway](https://github.com/agentgateway/agentgateway)| 4,812 at c=1 | 24,489 at c=256 | 5.1× / 256× | c=16 | 248 µs → 44.3 ms | c=1024 | c=1024 |
38
-
|[AISIX (api7)](https://github.com/api7/aisix)| 4,107 at c=1 | 16,936 at c=16 | 4.1× / 16× | c=8 | 270 µs → 4.5 s | none | c=32768 |
39
-
|[Helicone](https://github.com/Helicone/ai-gateway)| 3,329 at c=1 | 15,091 at c=32 | 4.5× / 32× | c=8 | 323 µs → 16.2 ms | c=128 | c=128 |
40
-
|[Kong](https://github.com/Kong/kong)| 4,321 at c=1 | 22,593 at c=256 | 5.2× / 256× | c=64 | 419 µs → 104 ms | c=1024 | c=2048 |
30
+
|[One-API](https://github.com/songquanpeng/one-api)| 29 at c=1 | 37 at c=16 | 1.3× / 16× | c=16 | 42.7 ms → 2.96 s | c=8 | c=16 |
41
31
42
32
A rung that failed a request it had accepted qualifies for **no** frontier reading at any bound, so rate measured at or above the failing concurrency is not throughput the board will publish - the climb chart rules that region off. **none** in that column is a measured result across the whole ladder, not a missing one.
43
33
@@ -47,37 +37,33 @@ Same box, same mock, one gateway at a time. Streaming figures are the overhead t
**✕** cells are measured refusals, not gaps: the gateway was offered the load and could not do the thing (buffered instead of streaming, rejected the Anthropic shape, or has no native key/limit governance). **n/a** = that suite hasn't been run for this gateway yet.
Method: added latency = gateway p99 − direct-to-mock p99 at concurrency 1; a frontier reading = the highest req/s any probed concurrency carried while 99% of requests finished under the STATED bound and the gateway failed none it accepted (readings are published at 1, 5, 10, 50, 100 ms and with no bound; the columns above use 10 ms, and every caption names the bound it used); cost figures divide that 10 ms reading by $0.1632/hr for the pinned 4-core (m7g.xlarge) slice; RSS idle = after first 200, steady state = the level the RSS settled at under load. Same box, same mock, same load, one gateway at a time. Each gateway's source ref is pinned in its own `gateways/<name>/definition.json`; the built commit is in each row.
82
68
83
-
<sub>Page + charts regenerated **2026-07-30 04:52 UTC** from the raw `results/*.json`.</sub>
69
+
<sub>Page + charts regenerated **2026-07-30 05:18 UTC** from the raw `results/*.json`.</sub>
0 commit comments