Skip to content

Support use vLLM DP + EP in async engine #1101

Open
#2517
Open
Support use vLLM DP + EP in async engine#1101
#2517
Assignees
yuki-97guyueh1
Labels
FeatureSpeedvllm

Description

@yuki-97
yuki-97
opened on Sep 9, 2025

#1081 supports using vLLM DP + EP in sync engine.

vLLM's DP + EP is supported for async engine for serve mode, but has issues for offline mode (which we used).
We need to hack something in vLLM to support that.

Ideally, after vLLM's FP8 inference for MoE model and DeepEP supported, we can get some speedup from DP + EP, and we need to bring the benefit to async RL / async rollout which needs to use vLLM async engine.

Activity

  1. yuki-97
    added
    vllm
    on Sep 9, 2025
  2. yuki-97
    self-assigned this
    on Sep 9, 2025

  3. yuki-97 commented on Nov 5, 2025

    @yuki-97
    yuki-97
    on Nov 5, 2025
    ContributorAuthor

    DP+EP+TP or DP+EP+TP+PP could work for async vLLM engine if EP == TP, which means we won't use vLLM DP.

    But if EP != TP, there will be two issues seperately.

    Repro:

    # DP + EP, no TP/PP
NRL_FORCE_REBUILD_VENVS=true \
NCCL_NVLS_ENABLE=0 \
uv run python examples/run_eval.py \
    generation.model_name=Qwen/Qwen3-30B-A3B-Instruct-2507 \
    generation.vllm_cfg.async_engine=true \
    generation.vllm_cfg.expert_parallel_size=2 \
    cluster.num_nodes=1 \
    cluster.gpus_per_node=8

# DP + EP + TP/PP, and EP != TP
    generation.vllm_cfg.pipeline_parallel_size=2 \
    generation.vllm_cfg.expert_parallel_size=2 \
OR
    generation.vllm_cfg.tensor_parallel_size=2 \
    generation.vllm_cfg.expert_parallel_size=4 \

    Error Log:

    # DP + EP, no TP/PP
(VllmAsyncGenerationWorker pid=2506931)   File "/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_nemorl/users/yukih/reinforcer/nemo_rl/models/generation/vllm/vllm_worker.py", line 328, in __init__
(VllmAsyncGenerationWorker pid=2506931)     self._create_engine(llm_kwargs)
(VllmAsyncGenerationWorker pid=2506931)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_nemorl/users/yukih/reinforcer/nemo_rl/models/generation/vllm/vllm_worker_async.py", line 140, in _create_engine
(VllmAsyncGenerationWorker pid=2506931)     self.llm = AsyncLLM.from_engine_args(self.llm_async_engine_args)
(VllmAsyncGenerationWorker pid=2506931)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 235, in from_engine_args
(VllmAsyncGenerationWorker pid=2506931)     return cls(
(VllmAsyncGenerationWorker pid=2506931)            ^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 134, in __init__
(VllmAsyncGenerationWorker pid=2506931)     self.engine_core = EngineCoreClient.make_async_mp_client(
(VllmAsyncGenerationWorker pid=2506931)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 101, in make_async_mp_client
(VllmAsyncGenerationWorker pid=2506931)     return DPLBAsyncMPClient(*client_args)
(VllmAsyncGenerationWorker pid=2506931)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1125, in __init__
(VllmAsyncGenerationWorker pid=2506931)     super().__init__(vllm_config, executor_class, log_stats,
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 975, in __init__
(VllmAsyncGenerationWorker pid=2506931)     super().__init__(vllm_config, executor_class, log_stats,
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 769, in __init__
(VllmAsyncGenerationWorker pid=2506931)     super().__init__(
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 448, in __init__
(VllmAsyncGenerationWorker pid=2506931)     with launch_core_engines(vllm_config, executor_class,
(VllmAsyncGenerationWorker pid=2506931)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/root/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 137, in __enter__
(VllmAsyncGenerationWorker pid=2506931)     return next(self.gen)
(VllmAsyncGenerationWorker pid=2506931)            ^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 715, in launch_core_engines
(VllmAsyncGenerationWorker pid=2506931)     local_engine_manager = CoreEngineProcManager(
(VllmAsyncGenerationWorker pid=2506931)                            ^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 130, in __init__
(VllmAsyncGenerationWorker pid=2506931)     with set_device_control_env_var(
(VllmAsyncGenerationWorker pid=2506931)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/root/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 137, in __enter__
(VllmAsyncGenerationWorker pid=2506931)     return next(self.gen)
(VllmAsyncGenerationWorker pid=2506931)            ^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 168, in set_device_control_env_var
(VllmAsyncGenerationWorker pid=2506931)     value = get_device_indices(evar, local_dp_rank, world_size)
(VllmAsyncGenerationWorker pid=2506931)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2506931)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 188, in get_device_indices
(VllmAsyncGenerationWorker pid=2506931)     raise Exception(f"Error setting {device_control_env_var}: "
(VllmAsyncGenerationWorker pid=2506931) Exception: Error setting CUDA_VISIBLE_DEVICES: local range: [1, 2) base value: "1"

# DP + EP + TP/PP, and EP != TP
(VllmAsyncGenerationWorker pid=2496662)   File "/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_nemorl/users/yukih/reinforcer/nemo_rl/models/generation/vllm/vllm_worker_async.py", line 140, in _create_engine
(VllmAsyncGenerationWorker pid=2496662)     self.llm = AsyncLLM.from_engine_args(self.llm_async_engine_args)
(VllmAsyncGenerationWorker pid=2496662)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2496662)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 235, in from_engine_args
(VllmAsyncGenerationWorker pid=2496662)     return cls(
(VllmAsyncGenerationWorker pid=2496662)            ^^^^
(VllmAsyncGenerationWorker pid=2496662)     self.engine_core = EngineCoreClient.make_async_mp_client(
(VllmAsyncGenerationWorker pid=2496662)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2496662)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 101, in make_async_mp_client
(VllmAsyncGenerationWorker pid=2496662)     return DPLBAsyncMPClient(*client_args)
(VllmAsyncGenerationWorker pid=2496662)     super().__init__(vllm_config, executor_class, log_stats,
(VllmAsyncGenerationWorker pid=2496662)     self._ensure_stats_update_task()
(VllmAsyncGenerationWorker pid=2496662)   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1001, in _ensure_stats_update_task
(VllmAsyncGenerationWorker pid=2496662)     assert self.stats_update_address is not None
(VllmAsyncGenerationWorker pid=2496662)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmAsyncGenerationWorker pid=2496662) AssertionError
  4. guyueh1
    self-assigned this
    on Nov 5, 2025
  5. clumsy
    added 2 commits that reference this issue on Nov 10, 2025

    fix: Support vLLM DP+EP in async engine via Ray-level data parallelism (

    0b1ec1d

    fix: Support vLLM DP+EP in async engine via Ray-level data parallelism (

    385027b

  6. guyueh1 commented on May 12, 2026

    @guyueh1
    guyueh1
    on May 12, 2026
    Contributor

    This is not planned anymore, right? @yuki-97
    I'm closing this for now, feel free to reopen if this is still in the roadmap

  7. guyueh1
    closed this as completedon May 12, 2026
  8. guyueh1
    reopened this on May 21, 2026
  9. linear-code
    added
    Speed
    Feature
    on May 21, 2026

  10. guyueh1 commented on May 21, 2026

    @guyueh1
    guyueh1
    on May 21, 2026
    Contributor

    Async engine for vLLM DP is fixed in #2517

  11. guyueh1
    linked a pull request that will close this issuefeat: fix the vLLM DP path #2517on May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

FeatureSpeedvllm

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions