feat(deepep-efa): vLLM DeepEP-V2 MoE all-to-all over EFA via NCCL-GIN GDAKI (GPU-initiated)

[dmvevents]

What

Adds a vLLM + DeepEP-V2 MoE all-to-all test case that routes the expert-parallel dispatch/combine over AWS EFA using the NCCL-GIN GDAKI transport (NCCL_GIN_TYPE=3, OFI_NCCL_GIN_GDAKI=1) — the GPU-initiated path where the GPU kernel posts the RDMA work requests itself, instead of handing them to a CPU proxy thread.

Placement: 3.test_cases/pytorch/vllm/deepep-v2-GDAKI-efa, parallel to the CPU-proxy sample 3.test_cases/pytorch/vllm/deepep-v2-efa (PR #1230). It mirrors that sibling's layout, README section order, and voice; the only functional deltas are the GDAKI transport stack. (The proxy sibling is not on main yet — #1230 is open — so this PR branches from main and adds the GDAKI sample as a peer; both touch only 3.test_cases/pytorch/vllm/README.md's index table.)

Why GDAKI needs a different substrate

DeepEP's default transport is NVSHMEM/IBGDA, which EFA does not provide. The proxy sibling runs V2's ElasticBuffer over the GIN CPU-proxy. This sample uses GDAKI (kernel-posted WQEs), which on EFA needs a newer transport substrate, all built from public source and pinned by immutable SHA:

• rdma-core @ master post-PR#1701 (merged 2026-08-04) — EFA completion-counter verbs. No release tag carries it yet → SHA pin.
• libfabric @ main post-PR#12591 (merged 2026-07-28) — prov/efa consumes the comp-cntr caps; built against the rdma-core above.
• aws-ofi-nccl @ a3d2680 --enable-gdaki + a PR#1351 cherry-pick (forced-PCIe gdrcopy override). a3d2680 carries PR#1311 (the per-platform EFA hw-counter tristate OFI_NCCL_GDAKI_EFA_HW_COUNTER) and the GIN seq-space aliasing fix.
• DeepEP b306af06 + PR#612 (same base+PR as the proxy sample) + a dlsym-guarded V13 host-UC shim that is inert (logs a fallback and no-ops) unless the plugin exports a hook upstream does not — carried as evidence lineage, not a live code path.

Everything else — EP_REUSE_NCCL_COMM=0 (or serve init segfaults under vLLM's lazy NCCL comm), the vLLM wheel-pin, and the eager/non-eager fix stack — is identical to the proxy sample.

Measured evidence (2026-08-14, 2×/4× p5en.48xlarge H200, Qwen/Qwen3-30B-A3B-FP8)

Full tables + provenance + honest caveats in benchmarks/README.md.

• Eager + non-eager serve, DP16/EP16 and DP32/EP32 — every sweep 121/121 HTTP 200, zero GDAKI-CQE … status 9 (the known upstream failure signature) through every phase.
• Same-node-set, same-image transport A/B (env-flip only): GDAKI ≥ CPU-proxy in all 10 paired comparisons across the EP16+EP32 2×2, +0.2–3.2% aggregate tok/s.
• Honest scope: single sweep per arm (the 2×2 consistency shows a direction, not a tight interval); the validation nodes ran efa.ko 3.0.x/3.1.x, so cross-node proof is functional (efa-direct boot banner ×8 + coherent EP output), not a byte-level /sys hw-counter tally — that needs efa.ko ≥ 3.3.0 (documented in the node preconditions).

Build / reproducibility

• NGC-from-scratch: FROM nvcr.io/nvidia/cuda:13.0.0-devel-ubuntu22.04; every layer from a public source (EFA installer, rdma-core, libfabric, gdrcopy, torch cu130, aws-ofi-nccl, DeepEP, vLLM wheel). A tester with docker + NGC + GitHub/PyPI reproduces the image with zero private access.
• No registry is hardcoded — setup/env_vars.example uses the standard placeholder; the manifest uses REPLACE_WITH_YOUR_REGISTRY.
• setup_deepep_v2_gdaki_efa.sh is COPY'd (in-tree, reviewable), not curled, and is first-party for the GDAKI path — deliberately outside .github/workflows/deepep-vendor-sync.yml (that CI gates the NVSHMEM setup_deepep_efa.sh vendored copy, a different script). The V13 shim patch was dry-run-verified to apply cleanly (patch -p0, exit 0) against upstream DeepEP b306af06.

Layout

deepep-v2-GDAKI-efa/
  Dockerfile                       NGC-from-scratch; GDAKI stack + rdma-core/libfabric build layers
  setup_deepep_v2_gdaki_efa.sh     COPY'd: builds aws-ofi-nccl --enable-gdaki + stages DeepEP + V13 shim
  v13-host-uc-shim.patch           dlsym-guarded, inert on upstream
  setup/                           build-push.sh + env_vars.example (bring your own ECR)
  recipe/                          serve.sh, run-kernel-test.sh, build_deepep.sh (in-pod _C.so),
                                   verify-image.sh, benchmark.sh, benchmark_probe.py, apply-noneager-fix-stack.sh
  kubernetes/                      2-node StatefulSet + headless service (GDAKI env contract + hw-counter tristate)
  benchmarks/README.md             measured eager/non-eager/A-B tables (EP16 + EP32) + caveats

Opened as draft — pending maintainer review of placement/scope alongside #1230.

DCO signed off.

[KeitaW]

Let's incorporate necessary changes into upstream instead of patches.

[dmvevents]

Re-scoped per the repo's no-patches rule — thank you for stating it plainly rather than letting it surface in review.

What changed (this push):

• recipe/apply-noneager-fix-stack.sh (cherry-pick + build-time source rewrite) is removed. The case now ships zero patches.
• Eager (--enforce-eager) is the shipped + supported path — it is also our production recommendation today.
• Default compilation (CUDA graphs) is documented as pending one upstream fix: the empty-ExpertTokensMetadata guard, now filed as vllm-project/vllm#52632. Once it merges, a vLLM pin bump enables non-eager here with no recipe change — exactly the pin-bump-survivable shape the rule requires.
• The non-eager numbers in benchmarks/ are retained but relabeled as historical measurements taken with the then-unmerged guard.

This keeps the case alive across pin bumps by construction: nothing in the build rewrites upstream source.

[dmvevents]

One follow-up commit (6fe5b02b): the NCCL linker-precedence assert in the Dockerfile used a pattern ldconfig -p can never print (name => path — the real format is name (libc6,x86-64) => path), so the step failed 100% of the time. Found by live-building the sibling vLLM DeepEP-V2 case (#1230) from its branch, where the same assert shipped; the eager path there is now proven end-to-end on 2× H200 (details on #1230). Same one-line fix here; still zero patches.

[dmvevents]

@KeitaW the changes-requested items are addressed on this branch (patch machinery deleted, zero-patch eager path is the shipped recipe — details in the two comments above), and the sibling #1230 eager path is live-verified E2E from its branch. Ready for another look whenever convenient. (Noting it here since fork authors can't use the re-request-review button on this repo.)