🐛 [Bug] [NGC] L2 dynamo test failure on ORIN #3982
Description
Segmentation fault on ORIN-
python test_weight_streaming.py
TRT 10.11.0.33
PyTorch torch-2.10.0a0+a36e1d39eb
025-12-24T20:33:25.804448Z 01E Fatal Python error: Segmentation fault
2025-12-24T20:33:25.804455Z 01E
2025-12-24T20:33:25.804456Z 01E Thread 0x0000fffe8b81f180 (most recent call first):
2025-12-24T20:33:25.804458Z 01E File "/usr/lib/python3.12/threading.py", line 359 in wait
2025-12-24T20:33:25.804459Z 01E File "/usr/lib/python3.12/threading.py", line 655 in wait
2025-12-24T20:33:25.804461Z 01E File "/usr/local/lib/python3.12/dist-packages/tqdm/_monitor.py", line 60 in run
2025-12-24T20:33:25.804462Z 01E File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
2025-12-24T20:33:25.804464Z 01E File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap
2025-12-24T20:33:25.804465Z 01E
2025-12-24T20:33:25.804466Z 01E Thread 0x0000ffff8f67f180 (most recent call first):
2025-12-24T20:33:25.804468Z 01E File "/usr/lib/python3.12/socket.py", line 295 in accept
2025-12-24T20:33:25.804469Z 01E File "/usr/local/lib/python3.12/dist-packages/pytest_rerunfailures.py", line 466 in run_server
2025-12-24T20:33:25.804471Z 01E File "/usr/lib/python3.12/threading.py", line 1010 in run
2025-12-24T20:33:25.804472Z 01E File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
2025-12-24T20:33:25.804474Z 01E File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap
2025-12-24T20:33:25.804475Z 01E
2025-12-24T20:33:25.804476Z 01E Current thread 0x0000ffff91859960 (most recent call first):
2025-12-24T20:33:25.804477Z 01E File "/usr/local/lib/python3.12/dist-packages/torch/cuda/graphs.py", line 146 in reset
2025-12-24T20:33:25.804537Z 01E File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/runtime/_CudaGraphsTorchTensorRTModule.py", line 110 in _reset_captured_graph
2025-12-24T20:33:25.804540Z 01E File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/runtime/_CudaGraphsTorchTensorRTModule.py", line 127 in __del__
2025-12-24T20:33:25.804542Z 01E File "/opt/pytorch/torch_tensorrt/tests/py/dynamo/runtime/test_004_weight_streaming.py", line 401 in test_runtime_state_change
2025-12-24T20:33:25.804544Z 01E File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620 in standalone_func
2025-12-24T20:33:25.804546Z 01E File "/usr/local/lib/python3.12/dist-packages/torch/testing/_internal/common_utils.py", line 3325 in wrapper
2025-12-24T20:33:25.804547Z 01E File "/usr/lib/python3.12/unittest/case.py", line 589 in _callTestMethod
2025-12-24T20:33:25.804549Z 01E File "/usr/lib/python3.12/unittest/case.py", line 634 in run
2025-12-24T20:33:25.804550Z 01E File "/usr/local/lib/python3.12/dist-packages/torch/testing/_internal/common_utils.py", line 3477 in _run_custom
2025-12-24T20:33:25.804552Z 01E File "/usr/local/lib/python3.12/dist-packages/torch/testing/_internal/common_utils.py", line 3507 in run
2025-12-24T20:33:25.804554Z 01E File "/usr/lib/python3.12/unittest/case.py", line 690 in __call__
2025-12-24T20:33:25.804555Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/unittest.py", line 389 in runtest
2025-12-24T20:33:25.804557Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/runner.py", line 179 in pytest_runtest_call
2025-12-24T20:33:25.804559Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_callers.py", line 121 in _multicall
2025-12-24T20:33:25.804560Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_manager.py", line 120 in _hookexec
2025-12-24T20:33:25.804562Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_hooks.py", line 512 in __call__
2025-12-24T20:33:25.804564Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/runner.py", line 245 in <lambda>
2025-12-24T20:33:25.804565Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/runner.py", line 353 in from_call
2025-12-24T20:33:25.804604Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/runner.py", line 244 in call_and_report
2025-12-24T20:33:25.804606Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/runner.py", line 137 in runtestprotocol
2025-12-24T20:33:25.804608Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
2025-12-24T20:33:25.805244Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_callers.py", line 121 in _multicall
2025-12-24T20:33:25.805248Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_manager.py", line 120 in _hookexec
2025-12-24T20:33:25.805250Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_hooks.py", line 512 in __call__
2025-12-24T20:33:25.805252Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/main.py", line 396 in pytest_runtestloop
2025-12-24T20:33:25.805254Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_callers.py", line 121 in _multicall
2025-12-24T20:33:25.805256Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_manager.py", line 120 in _hookexec
2025-12-24T20:33:25.805257Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_hooks.py", line 512 in __call__
2025-12-24T20:33:25.805259Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/main.py", line 372 in _main
2025-12-24T20:33:25.805260Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/main.py", line 318 in wrap_session
2025-12-24T20:33:25.805262Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/main.py", line 365 in pytest_cmdline_main
2025-12-24T20:33:25.805264Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_callers.py", line 121 in _multicall
2025-12-24T20:33:25.805265Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_manager.py", line 120 in _hookexec
2025-12-24T20:33:25.805267Z 01E File "/usr/local/lib/python3.12/dist-packages/pluggy/_hooks.py", line 512 in __call__
2025-12-24T20:33:25.805268Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/config/__init__.py", line 199 in main
2025-12-24T20:33:25.805270Z 01E File "/usr/local/lib/python3.12/dist-packages/_pytest/config/__init__.py", line 223 in console_main
2025-12-24T20:33:25.805272Z 01E File "/usr/local/lib/python3.12/dist-packages/pytest/__main__.py", line 9 in <module>
2025-12-24T20:33:25.805273Z 01E File "<frozen runpy>", line 88 in _run_code
2025-12-24T20:33:25.805275Z 01E File "<frozen runpy>", line 198 in _run_module_as_main
2025-12-24T20:33:25.805276Z 01E
2025-12-24T20:33:25.805277Z 01E Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, PIL._imaging, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, psutil._psutil_linux, cython.cimports.libc.math, Cython.Utils, Cython.Plex.Actions, Cython.Plex.Transitions, Cython.Plex.Machines, Cython.Plex.DFA, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.StringIOTree, Cython.Compiler.Code, scipy._lib._ccallback_c, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, numba.mviewbuf (total: 45)
2025-12-24T20:33:31.281330Z 01E ./test.sh: line 43: 212 Segmentation fault (core dumped) python -m pytest -ra --ignore runtime/test_001_* --ignore runtime/test_000_* runtime/*
2025-12-24T20:33:31.216978Z 01E ++ RV=139
2025-12-24T20:33:31.218371Z 01E ++ [[ -z 139 ]]
2025-12-24T20:33:31.218374Z 01E ++ set +x
2025-12-24T20:33:32.390000Z 01E /mnt/cifs/home/swdl-fw-infra/gitlab-runner/builds/jobs/248189569/.custom_executor_scripts/build_script: line 234: 2395420 Hangup sleep ${TEST_TIMEOUT:-21600}
2025-12-24T20:33:32.500400Z 01E srun: error: a1u2n2g-0958-02: task 0: Exited with exit code 139
2025-12-24T20:33:32.125000Z 01O runtime/test_004_weight_streaming.py