Repository navigation
Install on GH200 - Torch not compiled with CUDA enabled #532
Description
I follow standard installation process as described:
uv venv
then
uv run python examples/run_grpo_math.py
Setup:
- 1x GH200
NVCC:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Jan_15_19:21:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.61
Build cuda_12.8.r12.8/compiler.35404655_0
Python: 3.12.11
During run I am getting:
File "/net/rkinas/nvidia/RL/venvs/nemo_rl.models.generation.vllm.VllmGenerationWorker/lib/python3.12/site-packages/torch/cuda/__init
__.py", line 363, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
torch.cuda.is_available()
False
When i install torch in different (clean) venv using
uv pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
cuda is available
Activity
terrykong commentedon Jun 21, 2025
Hi @rkinas . Thanks for the interest in NeMo RL and raising this. We eventually want to better support GH.
I've tested #533 on a single GH200 and it seems to work when running a simple loop like
where
vllmisn't used. You're welcome to try that out, but I think more work needs to be done to fully support all isolated environments (vllm/mcore).rkinas commentedon Jun 21, 2025
Hi @terrykong,
my goal is to setup GRPO training on 4-8 GH200 nodes (total 32-64 GPUs) - should be ok for 12B model. Now I am trying to setup Nemo-RL step by step (from one GPU to multinode) and learn it.
I run a training with fix you provided. It work for SFT. Thank you!
Unfortunately you are right :) GRPO does not work - it requires more changes.
(VllmGenerationWorker pid=664974) ValueError: Model architectures ['Qwen2ForCausalLM'] failed to be inspected. Please check the logs for more details.
(VllmGenerationWorker pid=664974) ERROR 06-21 16:02:51 [registry.py:363] ModuleNotFoundError: No module named 'vllm.vllm_flash_attn.layers'
snowmanwwg commentedon Jul 24, 2025
@rkinas did you get this run with VeRL? Would be curious to find out.
qiaochuz-nv commentedon May 5, 2026
QA RCCA Analysis
RCCA Status: Done
RCCA Category: covered
Test Action: none
Fix PR: #533, #1334
Root Cause: fix PR includes upstream test coverage.
Trigger Config: Install on GH200 - Torch not compiled with CUDA enabled
Coverage Assessment:
Conclusion: RCCA classification completed for this closed NeMo-RL GitHub issue.