Skip to content

Commit 0f23134

Browse files
devalshahamddevalshahamd
authored
Merge branch 'main' into main
2 parents 1393529 + 840c7c2 commit 0f23134

1,510 files changed

Lines changed: 103746 additions & 17796 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/check-api-compat/SKILL.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
---
2+
name: check-api-compat
3+
description: Determine the correct behavior of vLLM's OpenAI-compatible or Anthropic-compatible API by checking the official upstream spec, instead of guessing from vLLM's own code. Use for any bug, issue, PR, review, or question touching /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings, /v1/messages, or their request/response models — including "is this a bug or intended", "what should this field default to", "does upstream allow this", and any change that adds, removes, or renames a field on those models. Do not use for vLLM-internal APIs (engine, scheduler, LLM class) or for endpoints with no upstream counterpart.
4+
---
5+
6+
# Checking vLLM API compatibility against the upstream spec
7+
8+
vLLM's OpenAI and Anthropic server surfaces are *reimplementations* of someone
9+
else's spec. vLLM's own code, tests, and docstrings are not evidence of correct
10+
behavior — they are the thing under test. Establish what upstream says first,
11+
then compare.
12+
13+
Never answer "what should this field do" from memory. Field names, types, and
14+
especially defaults drift between spec versions.
15+
16+
## 1. Pin down the question
17+
18+
Name three things before looking anything up:
19+
20+
- **Which surface** — OpenAI-compatible or Anthropic-compatible. They are separate
21+
code paths in vLLM and separate specs.
22+
- **Which endpoint**, by path.
23+
- **Which field or behavior** — a request field, a response field, a streaming
24+
event shape, an error code, or a status code.
25+
26+
## 2. Read vLLM's implementation
27+
28+
In a vLLM checkout, the request/response models live beside each endpoint:
29+
30+
| surface | path |
31+
| --- | --- |
32+
| OpenAI chat | `vllm/entrypoints/openai/chat_completion/` |
33+
| OpenAI completions | `vllm/entrypoints/openai/completion/` |
34+
| OpenAI responses | `vllm/entrypoints/openai/responses/` |
35+
| OpenAI embeddings | `vllm/entrypoints/pooling/embed/` |
36+
| OpenAI shared base models, errors, usage | `vllm/entrypoints/serve/engine/protocol.py` |
37+
| Anthropic messages | `vllm/entrypoints/anthropic/` |
38+
| audio | `vllm/entrypoints/speech_to_text/*/protocol.py` |
39+
40+
Each directory holds `protocol.py` (the Pydantic models), `serving.py` (behavior),
41+
and `api_router.py` (routing and status codes). Read the field declaration *and*
42+
the code that consumes it — a field can be declared faithfully and then ignored,
43+
which is still an incompatibility.
44+
45+
## 3. Fetch the upstream spec
46+
47+
Prefer a source that can actually be read as text, in this order.
48+
49+
### OpenAI
50+
51+
1. The OpenAPI spec — authoritative, machine-readable, and fetches cleanly:
52+
`https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml`
53+
2. The installed `openai` Python SDK's typed params, which track the spec closely.
54+
Locate them with `python -c "import openai,os;print(os.path.dirname(openai.__file__))"`,
55+
then read e.g. `types/chat/completion_create_params.py`,
56+
`types/chat/chat_completion.py`, `types/chat/chat_completion_chunk.py`,
57+
`types/responses/`. Note the SDK version — an old SDK is an old spec.
58+
3. The prose reference at `https://platform.openai.com/docs/api-reference/...`
59+
for semantics the types do not carry. It is script-heavy and often fetches
60+
poorly; treat a failed fetch as a fetch failure, not as absence of a field.
61+
62+
### Anthropic
63+
64+
1. `https://docs.claude.com/en/api/messages` and
65+
`https://docs.claude.com/en/api/messages-streaming` — these render as clean
66+
text and are the primary source for both fields and streaming event order.
67+
2. The installed `anthropic` SDK's `types/message_create_params.py`,
68+
`types/message.py`, `types/raw_message_stream_event.py`.
69+
70+
Say which source and which version answered the question, so the finding can be
71+
re-checked later.
72+
73+
## 4. Compare on all four axes
74+
75+
A field is compatible only when **name**, **type**, **default**, and **semantics**
76+
all match. Most real bugs hide in the last two: a field that exists with the right
77+
type but the wrong default, or one that is accepted and silently dropped.
78+
79+
Also check, when relevant:
80+
81+
- **Streaming** — chunk shape, the order of events, and the terminal event
82+
(`data: [DONE]` for Chat Completions and Completions; `response.completed`,
83+
`response.failed`, or `response.incomplete` with no `[DONE]` for Responses;
84+
`message_stop` for Anthropic).
85+
- **Errors** — HTTP status *and* the error body's `type`/`code` strings.
86+
- **Omitted vs null** — upstream often distinguishes an absent field from an
87+
explicit `null`. Pydantic defaults erase that distinction unless the model is
88+
written to preserve it.
89+
90+
## 5. Classify before calling it a bug
91+
92+
Three outcomes, and they are not the same:
93+
94+
- **Matches upstream** — no action.
95+
- **vLLM extension** — a field upstream has no name for at all (`vllm_xargs`,
96+
`chat_template_kwargs`, sampling knobs like `min_p`). These are intentional and
97+
allowed. Do not report them as violations. Check per surface: `top_k` is an
98+
extension on OpenAI but a real upstream field on Anthropic Messages.
99+
- **Incompatibility** — a field that *shares a name with upstream* but differs in
100+
type, default, or behavior, or an upstream field that is missing, ignored, or
101+
malformed. This is the reportable case.
102+
103+
An unsupported-but-declared field is its own subcase: decide whether vLLM rejects
104+
it explicitly or accepts and ignores it, and report which, since silently ignoring
105+
a documented field is the more damaging failure.
106+
107+
## 6. Report
108+
109+
State: the endpoint and field, what upstream specifies (with the source), what
110+
vLLM does (with `file:line`), which of the three outcomes it is, and the smallest
111+
fix. When behavior turns on a spec detail that is easy to disbelieve, quote the
112+
upstream line rather than paraphrasing it.

‎.agents/skills/check-api-compat/agents/openai.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
interface:
2+
display_name: "Check API Compat"
3+
short_description: "Check vLLM's OpenAI/Anthropic-compatible API against the upstream spec."
4+
default_prompt: "Use the vLLM API compatibility skill to check this endpoint or field against the upstream OpenAI or Anthropic spec."

‎.agents/skills/kernel-microbenchmark/SKILL.md‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: kernel-microbenchmark
3-
description: Build, debug, and interpret vLLM GPU kernel microbenchmarks for CUDA, Triton, and CuteDSL, including CUPTI timing, correctness checks, generated-code inspection, multi-GPU measurements, and SOL sanity checks.
3+
description: Build, debug, and interpret vLLM GPU kernel microbenchmarks for CUDA, ROCm/HIP, Triton, and CuteDSL, including CUPTI timing, correctness checks, generated-code inspection, multi-GPU measurements, and SOL sanity checks.
44
---
55

66
# Kernel Microbenchmark
@@ -15,7 +15,7 @@ description: Build, debug, and interpret vLLM GPU kernel microbenchmarks for CUD
1515
4. Compare against a baseline and report enough metadata to reproduce the
1616
result: GPU, dtype, shape, command, branch/commit, and relevant env vars.
1717
5. Treat explanations as hypotheses until backed by an artifact: ablation,
18-
generated PTX/SASS, profiler output, or controlled benchmark.
18+
generated PTX/SASS or AMD ISA, profiler output, or controlled benchmark.
1919
6. If the result changes the conclusion, preserve the compact lesson in a note,
2020
comment, benchmark table, or final summary. For experiments, a short
2121
`Question / Change / Correctness / Result / Observation / Next` note is
@@ -25,6 +25,12 @@ description: Build, debug, and interpret vLLM GPU kernel microbenchmarks for CUD
2525

2626
- Use FlashInfer CUPTI timing by default, with CUDA graph and cold L2 cache:
2727
`from flashinfer.testing import bench_gpu_time_with_cupti`.
28+
- ROCm: CUPTI and FlashInfer do not exist. Use HIP graph replay over rotating
29+
buffers whose total footprint exceeds the last-level cache (e.g. 256 MB on
30+
MI355X), so every call reads cold from HBM, timed with HIP events — see
31+
[benchmarks/graph_replay_benchmark.py](benchmarks/graph_replay_benchmark.py)
32+
(platform-agnostic). Prefer this over `do_bench`, whose per-rep
33+
launch/event overhead inflates small/decode-size kernels.
2834
- For compute-heavy kernels, report TFLOPS with the FLOP formula in the
2935
benchmark. For memory-heavy kernels, report estimated bytes moved and GB/s.
3036
For mixed kernels, report the most honest metric available and call out the
@@ -38,6 +44,11 @@ description: Build, debug, and interpret vLLM GPU kernel microbenchmarks for CUD
3844
formulas, skipped work, sparsity, caching, and whether the baseline is doing
3945
the same operation.
4046
- Force compilation/autotuning before measuring compiled kernels.
47+
- Tune every arm of a comparison before trusting it. A variant's default or
48+
inherited config can be far from its optimum, so conclusions drawn from
49+
untuned variants — especially negative ones — are unreliable. A coarse
50+
sweep of the cheap knobs at a couple of representative shapes is enough;
51+
report the winning config with the numbers.
4152
- Seed inputs when correctness comparisons matter.
4253
- Keep metadata setup, plan construction, allocation, random input generation,
4354
and logging outside the timed region unless that overhead is the experiment.
@@ -98,11 +109,21 @@ throughput conventions.
98109
a two-node TP run as equivalent to a local NVLink-domain run without checking.
99110
- Stabilize GPU clocks or run enough untimed work to reach a steady state.
100111
Alternate candidate order so clock, thermal, and rank-skew effects are shared.
112+
- ROCm: capturing ProcessGroupNCCL collectives in HIP graphs trips the NCCL
113+
watchdog (HIP forbids querying an event recorded in a capturing stream;
114+
the watchdog aborts the process). `TORCH_NCCL_BLOCKING_WAIT=1` before
115+
`import torch` disables the watchdog — benchmark-only knob. Ask for backend
116+
`"nccl"`; torch routes it to RCCL automatically (there is no `"rccl"`
117+
backend string).
101118

102119
## Included Examples
103120

104121
- Use [benchmarks/cupti_microbenchmark.py](benchmarks/cupti_microbenchmark.py)
105122
as a minimal single-GPU FlashInfer CUPTI timing pattern.
123+
- Use
124+
[benchmarks/graph_replay_benchmark.py](benchmarks/graph_replay_benchmark.py)
125+
as a graph-replay timing pattern over rotating buffers sized past the
126+
last-level cache — the ROCm/HIP default, also usable on CUDA.
106127
- Use
107128
[benchmarks/multi_gpu_gemm_rs.py](benchmarks/multi_gpu_gemm_rs.py) as a
108129
minimal distributed CUDA-graph timing pattern.

‎.agents/skills/kernel-microbenchmark/benchmarks/graph_replay_benchmark.py‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3+
4+
"""Graph-replay microbenchmark with rotating weight buffers (CUDA and ROCm).
5+
6+
Platform-agnostic: HIP implements the torch.cuda.* API, including CUDAGraph.
7+
Pattern: capture one graph (HIP graph on ROCm) containing R*INNER calls
8+
cycling over R rotating weight buffers, then time replays with CUDA/HIP
9+
events. The rotating buffers keep the captured footprint above the GPU's
10+
last-level cache, so every call reads weights cold from HBM. Prefer this over
11+
do_bench on ROCm: do_bench's per-rep launch/event overhead inflates
12+
small/decode-size kernels, and its default flush buffer is only as large as
13+
the LLC — on a 256 MB LLC that evicts marginally and leaves the cache full of
14+
dirty flush-buffer lines. The same pattern extends to multi-GPU benchmarks
15+
(capture per rank, collectives in identical order every replay); see the
16+
Multi-GPU section of SKILL.md.
17+
"""
18+
19+
import statistics
20+
21+
import pandas as pd
22+
import torch
23+
24+
LLC_TARGET_BYTES = 400 * 1024**2 # above 256 MB MI355X LLC and B200 L2 (126 MB)
25+
INNER = 2 # repeat slots within one graph to amortize per-replay overhead
26+
REPLAYS = 10 # replays per round: makes each measurement large vs overhead
27+
ROUNDS = 5 # median across rounds: replay time is not always stable
28+
WARMUP = 5 # warmup replays to reach steady clocks before timing
29+
MATMUL_CASES = [
30+
("compute-bound", 4096, 4096, 4096),
31+
("decode-GEMM", 16, 16384, 8192),
32+
]
33+
34+
35+
def timeit_graph(calls, inner=INNER, replays=REPLAYS, rounds=ROUNDS):
36+
"""Time a list of per-slot callables as one graph (zero launch overhead);
37+
weights rotate across slots so they stay HBM-cold. Returns the median over
38+
rounds of per-round means, ms per call."""
39+
for c in calls:
40+
c()
41+
torch.accelerator.synchronize()
42+
g = torch.cuda.CUDAGraph()
43+
with torch.cuda.graph(g): # capture runs on a side stream
44+
for i in range(len(calls) * inner):
45+
calls[i % len(calls)]()
46+
torch.accelerator.synchronize()
47+
events = [(torch.cuda.Event(True), torch.cuda.Event(True)) for _ in range(rounds)]
48+
for _ in range(WARMUP):
49+
g.replay()
50+
# No sync between warmup and the first round: queued replays keep the GPU
51+
# busy, so each round's launch latency stays outside its event window.
52+
for s, e in events:
53+
s.record()
54+
for _ in range(replays):
55+
g.replay()
56+
e.record()
57+
torch.accelerator.synchronize()
58+
samples = [s.elapsed_time(e) / (replays * len(calls) * inner) for s, e in events]
59+
return statistics.median(samples) # ms per call
60+
61+
62+
def main() -> None:
63+
if not torch.accelerator.is_available():
64+
raise RuntimeError("No GPU visible (torch.accelerator covers HIP on ROCm).")
65+
66+
torch.set_default_device("cuda")
67+
torch.manual_seed(0)
68+
69+
rows = []
70+
for name, m, n, k in MATMUL_CASES:
71+
a = torch.randn(m, k, dtype=torch.bfloat16) # shared: producer-warm
72+
out = torch.empty(m, n, dtype=torch.bfloat16)
73+
b_bytes = k * n * 2
74+
r = max(2, -(-LLC_TARGET_BYTES // b_bytes)) # ceil
75+
bs = [torch.randn(k, n, dtype=torch.bfloat16) for _ in range(r)]
76+
77+
def make_call(i, a=a, bs=bs, out=out):
78+
def call():
79+
torch.mm(a, bs[i], out=out)
80+
81+
return call
82+
83+
# correctness before timing, per slot (the graph would freeze a wrong
84+
# result into every replay)
85+
for i in range(r):
86+
make_call(i)()
87+
torch.accelerator.synchronize()
88+
torch.testing.assert_close(out, torch.mm(a, bs[i]), atol=1e-1, rtol=1e-1)
89+
90+
us = timeit_graph([make_call(i) for i in range(r)]) * 1e3
91+
rows.append(
92+
{
93+
"case": name,
94+
"shape": f"{m}x{n}x{k}",
95+
"slots": r,
96+
"us": us,
97+
"tflops": 2 * m * n * k / (us * 1e6),
98+
"gbps": 2 * (m * k + k * n + m * n) / (us * 1e3),
99+
}
100+
)
101+
102+
df = pd.DataFrame(rows)
103+
print(df.to_string(index=False, float_format=lambda x: f"{x:.3f}"))
104+
105+
106+
if __name__ == "__main__":
107+
main()

0 commit comments

Comments
 (0)