Repository navigation
NemoRL venv is not isolated #1132
Description
Describe the bug
uv commands use non-isolated user pip outside venv.
Therefore if users A and B have triton 3.2.0 and 3.3.1 in their user pip, it won't be reinstalled which leads to inconsistent behaviour
Steps/Code to reproduce bug
- Do any run
- Activate env from nemo-rl folder via
source venvs/<some_name>/bin/activate which pipandpip freeze
Expected behavior
pip should be contained in venv directory
Additional context
Use uv venv --seed
Activity
bogdansalyp commentedon Sep 15, 2025
We had same setup running fine for user A and crashing for user B. Same machines, same NemoRL
.venvsetups with no differences inprintenvanduv pip freezeThen we noticed the
venvs/folder is created which uses user's pip instead of isolated one.terrykong commentedon Sep 15, 2025
Hi @bogdansalyp . We don't recommend using the
uv pipAPI since there are cases where you can end up with an environment that does not respect the lock file.Could you explain why you want to source the venvs? If you are using nemo-rl the driver scripts (e.g.,
run_grpo_math.py) are launched withuv runand we handle the venvs of the worker processes for you so you don't ever need to explicitly activate themterrykong commentedon Dec 5, 2025
@bogdansalyp I’ll close this issue but feel free to open if you continue to have issues.
One PR that just landed you may be interested in is this one #1491. There were internal requests to be able to support bare Python in the containers b/c in certain circumstances when there were network outages on the intranet, uv run would fail b/c it did a quick metadata check over the internet. With that change you should be able to use bare Python. In fact, you don’t have to source the venvs if you want to use an interpreter from a particular actor. There’s some documentation in that PR that should help.
qiaochuz-nv commentedon May 5, 2026
QA RCCA Analysis
RCCA Status: Done
RCCA Category: covered
Test Action: none
Fix PR: #1491
Root Cause: fix PR includes upstream test coverage.
Trigger Config: NemoRL venv is not isolated
Coverage Assessment:
Conclusion: RCCA classification completed for this closed NeMo-RL GitHub issue.