Skip to content

CLAUDE.md

Latest commit

 

History

History
100 lines (76 loc) · 33.4 KB

CLAUDE.md

File metadata and controls

100 lines (76 loc) · 33.4 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this repo is

Two overlapping things live here:

  1. A content repo for vLLM recipes — structured YAML at models/<hf_org>/<hf_repo>.yaml, one per model. The path mirrors HuggingFace so the site URL /<hf_org>/<hf_repo> matches huggingface.co/<hf_org>/<hf_repo>.
  2. A Next.js 15 site (App Router, React 19, Tailwind v4) that renders those YAMLs as an interactive command builder + static JSON API under public/. The older top-level Markdown directories (DeepSeek/, Qwen/, etc.) and mkdocs.yml are the legacy MkDocs site, kept for reference during migration.

The site is the canonical surface. New work goes into YAMLs + the Next.js app; the .md files are historical sources for the migrated YAMLs.

Commands

pnpm dev                                # Next dev server (do not kill or rebuild as part of a task)
pnpm build                              # Regenerates JSON API + fetches logos/HF dates, then next build
pnpm lint                               # next lint (ESLint)
node scripts/build-recipes-api.mjs      # Rebuild the JSON API only — fastest validation that all YAMLs parse
node scripts/fetch-provider-logos.mjs   # Re-download HF org avatars to public/providers/
node scripts/fetch-hf-dates.mjs         # Refresh public/hf-dates.json (HF createdAt per model)
bash scripts/hf-info.sh <org>/<repo>    # Pull config.json / params.json for a model (used when authoring recipes)

There is no unit-test suite. The build-recipes-api script is the de-facto validator: if it prints ✓ JSON API: N models, <strategy-count> strategies the YAML set is internally consistent.

Dev-server convention: don't kill or restart the dev server as part of a task, and don't run pnpm build for each change. Use node scripts/build-recipes-api.mjs for fast feedback on YAML edits; rely on Next's HMR for src/ edits.

Commit convention: always use git commit -s (sign-off). The upstream vllm-project/recipes repo enforces DCO via the probot/dco GitHub App — commits without a Signed-off-by: trailer fail CI and block the PR. There's no GitHub-side auto-fix; the trailer has to be in the commit itself.

Architecture

Data pipeline

models/<org>/<repo>.yaml  ──┐
strategies/*.yaml          ─┼──► src/lib/*.js  ─► Next.js pages + JSON API in public/
kv_store/*.yaml            ─┤       (SSG, no runtime DB)
taxonomy.yaml              ─┘
  • taxonomy.yaml — the controlled vocabulary: hardware profiles (GPUs/node + VRAM), tasks, serving strategy ids, plus the kv_offload block — the composing KV-offload options (Simple, LMCache), each { display_name, label?, description, args[], strategies?, install?, companion? }, read data-driven by command synthesis. strategies is a per-option serving-strategy allowlist (LMCache = single-node only); companion renders as an extra command tab (e.g. lmcache server) while the option is active.
  • strategies/*.yaml — the 9 serving strategies: single_node_{tp,tep,dep}, multi_node_{tp,tep,dep,tp_pp,tp_dp}, pd_cluster. Each defines the vLLM flags and env for that parallelism pattern. multi_node_tp_dp is the hybrid TP-per-node + DP-across-nodes layout (parallelism: tp_dp in the YAML → synthesis emits --tensor-parallel-size <gpus/node> --data-parallel-size <nodes> --data-parallel-size-local 1, worker node N = DP rank N).
  • kv_store/*.yaml — the Mooncake KV-store deployments kv_store_{distributed,centralized}_mooncake (deploy_type: kv_store_lb — router + master + optional store + N vLLM instances, each spanning the Nodes-row node count). They power the KV Offload row, not the Strategy row; loadStrategies() merges them into the same map (looked up by id everywhere) but the JSON API publishes them under public/kv_store/ so /strategies.json stays the serving vocabulary.
  • src/lib/recipes.js, strategies.js, taxonomy.js — filesystem readers that parse YAML once per process.
  • src/lib/command-synthesis.js — the core of the site. Pure functions that take (recipe, variantKey, strategyName, hwProfileId, features, strategies, taxonomy, advancedArgs, nodeCount) and return either a single vllm serve command, a pair of head+worker commands (multi-node), or a prefill/decode pair (pd_cluster). All parallel flag decisions, per-generation hardware overrides, and multi-node mp args (--nnodes/--node-rank/--master-addr for TP, --data-parallel-* for DEP) live here.
  • scripts/build-recipes-api.mjs — writes public/taxonomy.json, public/strategies/*.json, public/kv_store/*.json, and public/<hf_org>/<hf_repo>.json at build time so agents can consume recipes without the website.

Next.js App Router

  • src/app/page.js — homepage (provider grid, alphabetical).
  • src/app/[org]/page.js — org overview, task-grouped sections.
  • src/app/[org]/[repo]/page.js — the recipe detail page. SSG via generateStaticParams over all YAMLs. Renders header + <CommandBuilder> + guide markdown (via react-markdown, not MDX — MDX chokes on <think> and <64K literals in guide text).
  • src/app/[org]/layout.js — wraps every recipe/org page with the left <ModelSidebar>, sorted by HF release date desc.
  • src/components/recipes/CommandBuilder.jsx — the interactive piece. Client component. Holds state for variant, hwId, strategyOverride, nodeCount, features, advanced; syncs all of them to URL query params; renders Hardware / Variant / Strategy / KV Offload / Nodes / Features / Advanced rows + the sticky command block. Calls resolveCommand on every change. The KV Offload row: Off (default) / Simple (taxonomy-driven --kv-transfer-config appended to the current strategy; disabled under pd_cluster) / one merged Mooncake pill. Every option COMPOSES with the serving strategy — the KV layer is orthogonal to parallelism, the Strategy row is always in effect. Mooncake wraps the selected strategy in a kv_store_lb deployment shell (router + master + optional store around N instances, each instance running the serving strategy's exact command — TP/TEP/DEP, single- or multi-node — with the MooncakeStoreConnector args/env appended last-wins; synthesis exposes the pairing as servingStrategy, the JSON API as serving_strategy). Offered on every non-omni recipe by default, needs scalable NVIDIA/AMD hardware (isKvStoreBrandSupported — the transfer engine has no CPU/TPU build; the brand-keyed vllm.install resolves CUDA vs non-CUDA wheel and the kv YAML's min_vllm_version bumps the Install header while active); a recipe opts a GPU out via kv_cache_strategy_hardware: { <kv_store_id>: { <gpu_id>: unsupported } }; URL param kv_offload holds the topology's strategy id. A Mooncake pick reveals a Store Topology row (Distributed embedded vs Centralized standalone store — short label from the kv_store YAML; distributed is the first-click default) and an Instances input (URL kv_instances; default = the topology's default_instances: distributed 1 — a single embedded instance is a complete CPU-offload tier — centralized 2). The router only exists at ≥2 instances; at 1 the client target falls back to VLLM_HOST:8000 and cURL/Bench move to the vLLM tab. The Nodes row keeps meaning nodes PER INSTANCE — relabeled "Nodes / Instance" while Mooncake is active, tooltips multiply totals by the instance count; at >1 each instance shards like multi_node_tp (tabs named Head / Node 1, same as MultiNodeBlock; --master-addr is $HEAD_IP at 1 instance, $VLLM_INSTANCE_N when routed). Under pd_cluster Mooncake COMPOSES instead of replacing: PD keeps Prefill/Decode/Router, each role's Nixl config swaps (last-wins) for the kv_store YAML's pd.<role>.args MultiConnector (Nixl + MooncakeStoreConnector), both roles gain the Mooncake env + config heredoc, and Mooncake Master/(Store) tabs lead the PD block; the Instances row hides (PD pools size themselves). Interaction-expression conventions: the Strategy row description gains an "Each Mooncake instance runs this strategy." suffix while Mooncake is active (composition made explicit, no dimming — the strategy is genuinely in effect); the active KV option's description + Docs link render under the row (Strategy-row style); Store Topology / Instances / the spec-method row use ConfigRow nested (blue left rail = child of the pick above); command-block tabs go through the shared CommandTabs (ARIA tablist, arrow-key roving focus, dimmed step numbers whenever the tabs form a launch sequence — always for Mooncake/kv_store, PD only when composing, companions); cURL/Bench render disabled-with-tooltip (never hidden) on tabs clients don't connect to; disabled Pills wrap in a span so Radix tooltips still explain WHY (native disabled swallows pointer events); the "Verified on " badge yields to the config summary whenever any KV Offload option is active (verification covered the plain serving path); setting KV Offload back to Off resets kv_instances/kv_instance to topology defaults.

Recipe YAML schema (authoritative)

Top-level keys, in this order:

  • meta — title, slug, provider, description, date_added, date_updated, difficulty, tasks[], performance_headline, related_recipes[], optional default_hardware, optional hardware: { gpu_id: verified | unsupported }. date_added is the immutable day the recipe first entered the catalog and drives the homepage's "Latest recipes" ranking; date_updated records the latest material recipe change. Hardware is tri-state: verified = tested end-to-end (green ✓ badge); unsupported = not yet runnable on this hardware today — compat gap / missing kernel / upstream blocker that could lift later (pill rendered but disabled + struck through, tooltip says "Not yet supported"); absent = silent default, assumed to work. Don't use unsupported for "untested" — only when you've confirmed it currently fails. default_hardware is a single GPU profile id (b300, dgx_spark_gb10, …) — the hardware the page and the JSON API open on. Precedence: URL ?hardware= > the user's stored global preference (set only by an explicit pill click) > default_hardware > the pickDefaultHardware heuristic (B200 for Blackwell-constrained variants, else H200). A returning visitor's own GPUs therefore still win — this key only replaces the model-blind heuristic. It's validated and fail-open: an id that's unknown to the taxonomy, precision-incompatible with the resolved variant, marked unsupported, or a restricted profile the recipe doesn't declare in meta.hardware falls through silently to the heuristic. Authoring rule: set it to the hardware the guide's numbers, the performance_headline and the verified badge were taken on — not to the biggest GPU. Landing on a verified profile is what makes the command block's green "Verified on " badge the built-in explanation for the choice, so no extra banner is needed; pointing it at an unverified GPU loses that badge and is worse than leaving the key absent. Most important for the JSON API, which has no URL or localStorage — default_hardware is the only way an author steers recommended_command (everything else stays addressable under by_hardware).
  • model — model_id, min_vllm_version, architecture (dense or moe), parameter_count, active_parameters, context_length, base_args[], base_env{}, optional default_frontend: python | rust (defaults to Python). Set default_frontend: rust for recipes that recommend it. The selector is saved per recipe and in ?frontend=python|rust; command synthesis emits VLLM_USE_RUST_FRONTEND=1 for Rust and omits the variable for Python. Keep this variable in the frontend synthesis path; use default_frontend to express recipe defaults. The JSON API uses the same recipe default.
  • dependencies (optional) — list of { command, note?, optional?, brand?, install_modes? }. Rendered as the "extra install" block above vllm serve. Use this for DeepGEMM pins, vllm[audio], pinned transformers commits, vllm-omni source installs, etc. optional: true dims the entry and excludes it from the block's "Copy all" button (the user must copy it explicitly). brand: NVIDIA | AMD | Intel (or array) hides the entry when the current hardware pill is a different brand — use for CUDA-only kernels (xformers, DeepGEMM) or ROCm-only wheels. Omit brand for platform-agnostic deps (diffusers from source, transformers commits, etc.). install_modes gates the entry by Install tab, and absent means [pip] — the block is pip-scoped by default because the Docker image already bundles the pip installs, so repeating them beside docker run is noise. install_modes: [pip, docker] opts an entry into both tabs; reserve it for host-side prerequisites Docker cannot satisfy, i.e. a gated checkpoint hf download must place on disk before buildDockerRun can bind-mount it (MiniMax-H3). A pip install never wants this. Keep the runnable fetch step here rather than as a # comment on the command — the block's "Copy all" yields executable lines, a comment does not.
  • features — map of {feature_key: { description, args[], env?, label? }}. Naming convention: tool_calling, reasoning, spec_decoding (the unified key for MTP / Eagle3 / ERNIE-MTP — don't use mtp, it was renamed across all recipes), long_context (RoPE/YaRN context extension — see below). Each feature is a boolean toggle by default. Optional label overrides the pill text (default: title-cased key).
    • env{} — env vars set only while the feature is on. Emitted from buildEnv as the mirror of the feature args block in buildArgs (step 7): applied after every override layer so an enabled feature wins, and gated identically (strategy allowlist, Mooncake-companion skip, active mode) so a feature's env and args can never disagree about whether it's on. A modes entry may carry env too, and hardware_overrides.<key>.env REPLACES it the same way args does. Use it for features whose documented launch line is env + flags rather than flags alone — long_context needs VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1" wherever the target window exceeds the checkpoint's max_position_embeddings. Propagates to every rendering: single-node, head + each worker on multi-node (one env map per deployment), both PD roles, and the Docker -e flags.
    • long_context — the context-extension toggle, ALWAYS in opt_in_features: static YaRN holds its scaling factor constant regardless of input length, so leaving it on costs short-prompt quality, and the longer window inflates the KV budget. Args are --hf-overrides '<rope json>' + --max-model-len <extended>; because features emit last and dedupeArgs is last-wins, both cleanly shadow a --max-model-len already in base_args. Use --hf-overrides, never --rope-scaling — the latter is what most model cards still print, but it was removed from vLLM's ModelConfig by 0.26, and transformers v5 renamed the config field rope_scaling → rope_parameters (vLLM 0.29 reads hf_config.rope_parameters). So a card's --rope-scaling '{...}' translates to --hf-overrides '{"rope_parameters":{...}}', and on a VL/omni checkpoint the block nests under text_config (a top-level override silently misses the decoder's config). Only add this feature where the model card documents a specific extension config — don't invent a factor.
    • Single-select sub-modes — when one feature has several mutually exclusive methods (e.g. a model offering more than one speculative method: MTP / DSpark / DFlash), replace the feature's flat args[] with a modes map and it renders as a "pick one" segmented row under the feature toggle instead of a boolean. A mode contributes only args (the --speculative-config) — the method, never the served checkpoint. Shape: { description, default_mode?, modes: { <mode_key>: { label?, description?, args[], variants?, hardware_overrides?, hardware? } } }. Only the selected mode's args are emitted, so exactly one --speculative-config is ever produced — no dedupe conflict between sibling methods, and no disables_features gymnastics. default_mode picks the initial mode (else the first key). A mode may restrict itself to specific checkpoints with variants: [<variant_key>, …] — e.g. the dspark method only exists on the fused DSpark checkpoint, so FP8/NVFP4 offer MTP only (a mode with no variants is available everywhere). The row renders only when ≥2 modes are available for the current variant — a single available method is left to the boolean feature toggle (no redundant one-pill row, no disabled cross-axis buttons). The effective mode is resolved by resolveModeKey (variant + hardware availability, then user pick → variant.default_modes → feature.default_mode → first available). Keep the boolean args[] form for recipes that ship a single method — the two forms coexist and single-method recipes need no change. Selected mode syncs to the URL as ?fmode=spec_decoding:dflash (default omitted). Synthesis + UI helpers: defaultModeFor/isModeSupported in command-synthesis.js.
      • Method vs. checkpoint — the two axes stay separate. The served checkpoint (vllm serve <X>) is always owned by the Variant row (variant.model_id); a mode only picks the --speculative-config method. So: a method that adds an externally-trained draft (a separate speculator repo named inside the config model, e.g. GLM's RedHatAI/GLM-5.2-speculator.dspark-preview) is a pure mode — <X> is unchanged. A method that ships as a fused checkpoint you serve instead (the draft is baked in, --speculative-config carries no model, e.g. official deepseek-ai/DeepSeek-V4-Pro-DSpark) is a variant (its model_id) plus a mode for the config. Couple them with variant.default_modes: { <feature>: <mode> } (see variants below): selecting the DSpark variant flips spec_decoding to the dspark method by default, while the checkpoint still runs MTP if the user switches the mode back. Don't put --speculative-config in a variant's extra_args (last-wins dedupe would let the boolean/mode feature shadow it) — keep it in the mode.
      • Per-mode hardware gating — a mode may carry hardware_overrides.<gen>.args (per-generation arg swap, same semantics as feature-level) and a tri-state hardware: { <gen_or_gpu_id>: unsupported } to gate a mode to specific GPUs (e.g. a DFlash draft that only ships for Blackwell → hardware: { hopper: unsupported }); the pill is struck-through on incompatible hardware and the selection auto-falls-back to the first supported mode.
    • Per-strategy gating — a feature may declare strategies: [<strategy_id>, …] (allowlist; absent = every strategy). Gate a feature when its args collide with a strategy's own flags or its setup doesn't generalize (e.g. a --kv-transfer-config that would clobber pd_cluster's NixlConnector — features emit last, so last-wins dedupe would win). On excluded strategies the pill renders disabled ("Not available with this strategy"); the toggle state is preserved and synthesis independently skips the feature's args and companion. Helper: isFeatureAllowedForStrategy in command-synthesis.js. NB: KV offloading itself is NOT a per-recipe feature in this repo — it lives on the universal KV Offload row (taxonomy kv_offload + kv_store/), which reuses the same gating idea via isKvOffloadAllowedForStrategy.
    • Companion process — a feature may declare companion: { label, description?, command } for a helper that must run alongside vllm serve on the same node. While the feature is on, the single-node command block renders PD-style tabs (vLLM Serve · <label>); the companion tab shows the raw command (no docker wrap, no env prelude) with the description as a # comment line. Carried in the JSON API as companions[] on single-node renderings. The taxonomy's kv_offload.<key>.companion (LMCache's lmcache server) flows through the same machinery.
  • opt_in_features[] — subset of feature keys that default OFF; others default ON. Typically spec_decoding is opt-in. (For a modes-feature, opt-in still governs the on/off toggle; the mode row only appears once the feature is on.)
  • variants — must include a default variant. Each has precision, vram_minimum_gb, optional description, optional model_id override (for quantized checkpoints shipped under a different HF repo, e.g. nvidia/*-NVFP4), optional extra_args/extra_env, optional label (overrides the pill text, which otherwise shows precision.toUpperCase() — use it to disambiguate two variants of the same precision, e.g. a fused-spec checkpoint → label: "DSpark"), optional default_modes: { <feature>: <mode> } (steers a modes-feature's default mode when this variant is selected and auto-enables that feature even if it's in opt_in_features — e.g. the DSpark checkpoint variant turns spec_decoding on and defaults to the dspark method; see spec_decoding modes above), optional min_vllm_version / nightly_required (bump the recipe baseline for a variant that needs a newer vLLM — the Install block shows the higher version and, if nightly_required, switches to the nightly wheel index + tag; e.g. the DSpark checkpoint requires 0.25.0, currently nightly), optional tp (per-variant single-node TP override; either a bare number or a hardware-aware map keyed by exact GPU id, generation, brand, or default, e.g. { b300: 1, h200: 2 }. Use it when a variant has documented per-GPU layouts or stricter sharding constraints — typically FP8-block-quantized siblings whose moe_intermediate_size / TP must stay divisible by block_n=128. Do not put --tensor-parallel-size inside extra_args; it gets silently overridden by the strategy emitter), optional hardware_overrides with two disjoint keyspaces: a generation key (hopper/blackwell/amd/nvidia) layers ADDITIVELY on top of the recipe-level hardware_overrides.<gen> baseline — applies to every variant including default (unlike the top-level variant extra_args, which is a non-default checkpoint delta; the generation HO is hardware-conditional config, so default uses it too). Use it for a per-variant hw delta that only some precisions want: e.g. the FP8 checkpoints of DeepSeek-V4 add --moe-backend deep_gemm_mega_moe (an FP8-only MoE kernel) on Blackwell while the NVFP4 variant adds nothing — the recipe-level blackwell block holds only the shared --attention_config.use_fp4_indexer_cache=True. Precedence: a per-strategy strategy_overrides.<s>.hardware_overrides.<gen> still REPLACES both the recipe baseline and the variant delta (e.g. single_node_tp dropping the mega-kernel), so strategy HO > (recipe HO + variant HO). The other keyspace, an exact-GPU-id key (b200, mi355x, …), is additive too (non-default variants only) and can also carry vram_minimum_gb (GPU-resident fit budget after CPU weight offload — replaces the variant's full vram_minimum_gb for fitsSingleNode on that GPU only; never bump taxonomy vram_gb with host RAM), docker_image (per-variant-per-GPU image pin) and install: { pip: false } — the per-GPU form of model.install.pip: false, declaring that no wheel can run this recipe on that GPU so the Install block hides the pip tab there (and the command block renders docker run) while leaving pip intact on every other hardware. Use it when the pinned image carries kernels that aren't in vLLM main, so the generated args would abort on an official wheel — DeepSeek-V4-Flash's B12X flags on DGX Spark. Not for "docker is easier here": a pip path that merely isn't the recommended one stays offered.
    • precision_hardware (optional, variant-level) — overrides the default hardware constraint implied by precision, using { brand?, generation? }. Use only for exceptional checkpoints whose runtime fallback expands compatibility beyond the stored format, such as NVFP4 weights that vLLM dequantizes to BF16 on Hopper. Pair it with supported_hardware when only specific GPU profiles are documented.
  • compatible_strategies[] — subset of the serving strategy ids. Dense models typically only get single_node_tp + multi_node_tp; MoE models can have every serving strategy. Membership doubles as the default support state per GPU: listed = supported on every hardware, unlisted = supported nowhere — strategy_hardware overrides either default per hardware, and a strategy granted supported there joins the offered set without being listed here (appended after the listed ones; helper effectiveCompatibleStrategies in command-synthesis.js). Do NOT list kv_store_*_mooncake ids here — KV-offload options (Simple + both Mooncake modes) are implicit on every non-omni recipe and feed the "KV Offload" row, not the Strategy row.
  • strategy_min_gpus (optional) — minimum GPUs a strategy needs for this model. Either a bare number (one bar for every strategy) or a map keyed by strategy id ({ multi_node_dep: 16, multi_node_tp: 8 }) so one parallelism can demand more than another — e.g. Kimi-K3 runs TP/TEP from 8 GPUs but needs 16 for DEP. The map takes a second, disjoint keyspace: an exact GPU id (h100: 32, or nested per-strategy h100: { multi_node_dep: 32 }) layering a hardware-conditional floor on top of the strategy bars, highest wins. Use it when one GPU needs a bigger cluster than the recipe baseline — K3's ~1.68 TB of weights fits 2 H200 nodes but needs 4 on H100, so h100: 32, which also drops single_node_tp from the H100 pill row for free (8 GPUs can't clear a 32-GPU bar). Only exact GPU ids are read, never generations: GPUs of one generation differ in VRAM, which is exactly what moves the floor. Opt-in and fail-open: absent = no floor, so existing recipes are unaffected. A multi-node strategy stays offered and grows the node count to meet its bar when picked (the Nodes row gains the count that first clears it, e.g. 4 on a 4-GPU NVL tray); a single-node strategy can't scale, so it drops out when the floor exceeds one node's GPUs. Applies to PD pools too, keyed by pool parallelism (tp/tep/dep → the matching multi_node_* entry) — each pool is sized to its own floor. Helpers: minGpusForStrategy / nodesForStrategy / isStrategyReachable in command-synthesis.js, shared by the builder and the JSON API. Distinct from the (currently unread) hardware_match.min_gpus in strategies/*.yaml, which is a per-strategy structural minimum rather than a per-recipe one.
  • kv_cache_strategy_hardware (optional) — per-strategy/per-GPU opt-OUT for the Mooncake pills: { <kv_store_id>: { <gpu_id>: unsupported } }. Follows the repo's fail-open hardware convention: absent = assumed to work (pill enabled on any scalable GPU); unsupported disables the pill and makes the JSON API skip that strategy on that hardware. "Off" and "Simple" need no gating (Simple is defined once in taxonomy.yaml → kv_offload).
  • strategy_hardware (optional) — per-serving-strategy hardware overrides, { <strategy_id>: { <hw_key>: unsupported | supported } }, layered on the membership default from compatible_strategies (listed = supported everywhere, unlisted = nowhere). Each per-strategy map takes the variant-tp keyspace — exact GPU id > generation > brand (lowercase) > default, first key present wins, an explicit default: beating the membership baseline — so both directions are one line per hardware key: unsupported opts a listed strategy out of a GPU/generation/brand; supported opts an unlisted strategy in, which also adds it to the offered set (no compatible_strategies entry needed — e.g. Kimi-K3 offers multi_node_tp_dp only via { h200: supported, gb200: supported, gb300: supported }). unsupported renders the Strategy pill disabled with a tooltip, keeps it out of the recommended/active fallback, and makes the JSON API skip that strategy on that GPU. Use unsupported when a listed strategy is genuinely unusable on specific hardware (e.g. multi-node pure-TP on Hopper, which has no cross-node NVLink) — not for "untested". Helpers: isStrategySupportedOnHardware + effectiveCompatibleStrategies in command-synthesis.js (shared by the builder and the JSON API).
  • hardware_overrides — optional per-generation tweaks keyed by hopper, blackwell, amd, or the brand-wide nvidia. Generation-specific blocks take precedence over the NVIDIA fallback. Each supplies { extra_args[], extra_env{} }.
  • strategy_overrides — optional per-strategy tweaks (rare). For pd_cluster specifically: prefill/decode sub-objects set each pool's nodes, parallelism (tp|tep|dep), and tp; router.intra_node_data_parallel_size pins the router flag that otherwise derives from the pools' max(dp_local); plus optional pool_modes: [<mode>, …] — narrows the PD-pool parallelism pills INDEPENDENTLY of the Strategy row. PD-pool modes are otherwise derived from compatible_strategies[] suffixes (via pdPoolModes), so pool_modes is how you offer TP/TEP as standalone serving strategies while locking disaggregated PD to a subset (e.g. pool_modes: [dep] → both pools DEP-only, single mode hides the pills; synthesis snaps any resolved mode not in the set to the first allowed one, so the JSON API stays consistent too).
  • omni (optional) — vllm-omni online-serving config. Required when meta.tasks includes omni. Shape: { serve_binary?, port?, task_label?, task_hint?, tasks: [...] }. tasks accepts bare ids from the catalog (t2i, i2i, t2v, i2v, ti2v, t2a) or { id, key?, model_id?, vram_minimum_gb?, description?, extra_args?, curl?, benchmark? } overrides — Wan2.2 uses overrides to swap the served checkpoint per task. benchmark is a static command override for diffusion or other non-LLM endpoints that cannot use the default vllm bench serve command. serve_binary: "vllm-omni serve" swaps the binary; currently unused, and needed only for recipes pinned below vLLM 0.20.0, where the vllm console-script does not yet delegate to vllm-omni on --omni. Catalog lives at src/lib/omni-tasks.js. key is the unique selector and URL identity when multiple request presets share one catalog id; absent key falls back to id. task_label and task_hint customize that selector row.
  • guide — markdown string (|-block), rendered with react-markdown + remark-gfm + rehype-slug.

VRAM formula: vram_minimum_gb = ceil(params × bytes × 1.2). Bytes per param: bf16/fp16=2, fp8/int8/awq/gptq=1, int4/nvfp4/fp4/mxfp4=0.5. For MoE, use total params (inactive experts still live in VRAM). Exception — mixed-precision quants (NVFP4/ModelOpt, hf_quant_config.json → quant_algo: MIXED_PRECISION): only MLP linears are 4-bit while attention/KV/embeddings stay FP8+, so the table underestimates. Size from the real checkpoint instead: ceil(safetensors total_size_GB × 1.2).

Non-obvious design decisions

  • Hardware is never blocked by VRAM. Multi-node scales VRAM, so command-synthesis.js only uses VRAM as a display hint. Precision constraints (NVFP4/FP4 require Blackwell by default) are the only things that disable a hardware pill; exceptional runtime fallbacks can broaden them with variant-level precision_hardware.
  • Default hardware is always H200 (or B200 for Blackwell-constrained variants). pickDefaultHardware prefers NVIDIA over AMD; loadPreferences deliberately ignores AMD entries from localStorage so first-page-load is always H200.
  • Omni recipes render vllm serve --omni via a dedicated resolveOmniCommand path (no strategy / multi-node / pd). The command builder adds a Task pill row (T2I / I2I / T2V / I2V / TI2V / T2A) above Hardware, with task-specific cURL examples in the popover. Detection: meta.tasks includes omni. The --omni flag is auto-injected, so don't add it to model.base_args.
  • Multi-node uses vLLM mp (multiprocessing) backend, not Ray. For TP/TEP every node runs the same command varying --node-rank; rank>0 adds --headless. For DEP, Node N adds --data-parallel-start-rank = N × gpus_per_node. One command per node, never a repeated one: synthesis renders the head (rank 0) plus one command per follower rank via buildArgs(role, "worker", rank) — followerRanks(nodeCount) = [1 … nodeCount-1] — so the command block shows a tab per node (Head · Node 1 · … · Node N-1) and a 4-node cluster gets three distinct worker commands. Same in the Mooncake block (nodes per instance). The JSON API carries the full list as worker_commands[] / worker_argvs[] (+ worker_docker_*s[], and vllm_worker_*s[] under kv_store_lb); the older singular worker_command/worker_argv stay as rank 1 for compatibility — agents should loop the plural arrays, since replaying rank 1 on every node hangs the rendezvous.
  • Taxonomy hardware specs follow inferencex.semianalysis.com/gpu-specs: B200 SXM is 180 GB/GPU (not the nameplate 192, which is pre-ECC), GB200/GB300 are 4-GPU compute trays (NVL4, the NVL72 rack unit), B300 is 268 GB/GPU.
  • Legacy Markdown under DeepSeek/, Qwen/ etc. is read-only reference. When migrating or updating a recipe, read those to extract flags and env, but write the YAML; don't edit the Markdown.
  • public/ is generated — don't commit it. .gitignore should cover it; if something slips through, leave it untracked.

Contributing a new recipe

A dedicated skill lives at .claude/skills/add-recipe/SKILL.md and is auto-loaded when the user asks to add/create/contribute a recipe. It covers HF metadata extraction, the full schema, VRAM formula, provider-registry updates, and the validation/commit loop.