slight code reorg and bug correction for cross_compile #3472

apbose · 2025-04-14T20:34:51Z

Addresses the following for the cross_compile_for_windows feature-

Slight code reorg
replace cross_compile_flag with cross_compile_module
Bug fix for 🐛 [Bug] Wrong output shape from cross compiled exported program #3400

narendasan · 2025-04-14T22:33:09Z

py/torch_tensorrt/dynamo/_exporter.py

+                # insert the no_placeholder node in the graph which should be replaced to the actual execute_engine node while load in the windows
+                trt_node = gm.graph.call_function(
+                    torch.ops.tensorrt.no_op_placeholder_for_execute_engine.default,
+                    (trt_module_node.args, *engine_info),


Do we still need to unpack this list?

We would still need to unpack the list. Else while loading in windows it shows

File "C:\Users\abose\Documents\work\TensorRT\torchTRT\Lib\site-packages\torch\_export\serde\serialize.py", line 2258, in deserialize_inputs args.append(actual_args[schema_arg.name]) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^ KeyError: 'name'

narendasan · 2025-04-15T21:18:04Z

py/torch_tensorrt/runtime/_utils.py

@@ -144,6 +144,7 @@ def no_op_placeholder_for_execute_engine(
    serialized_hardware_compatible: str,
    serialized_metadata: str,
    serialized_target_platform: str,
+    serialized_require_output_allocator: str,


Move this placeholder op to runtime/meta_ops

narendasan · 2025-04-15T21:19:13Z

py/torch_tensorrt/dynamo/_exporter.py

-            getitem_nodes = trt_node.users
-            for idx, getitem_node in enumerate(getitem_nodes):
-                getitem_node.meta["val"] = trt_node.meta["val"][idx]
+        no_op_placeholder_node.replace_all_uses_with(trt_node)


Can you add a multi output testcase to the cross compile tests?

apbose · 2025-04-15T21:26:44Z

py/torch_tensorrt/dynamo/_exporter.py

+        no_op_placeholder_node.replace_all_uses_with(trt_node)
+        getitem_nodes = trt_node.users
+        for idx, getitem_node in enumerate(getitem_nodes):
+            getitem_node.meta["val"] = trt_node.meta["val"][idx]



@narendasan this is the part which should address the bug

narendasan · 2025-04-18T23:54:26Z

tests/py/dynamo/runtime/test_003_cross_compile_for_windows.py

@@ -22,6 +22,8 @@ class Add(torch.nn.Module):
            def forward(self, a, b):
                return torch.add(a, b)

+        print("here")


Remove this

narendasan

LGTM

apbose · 2025-04-21T18:33:21Z

@bowang007 on linux converter tests I see-
FAILED automatic_plugin/test_flashinfer_rmsnorm.py::TestAutomaticPlugin::test_rmsnorm_float_0 - RuntimeError: FlashInfer requires sm75+
and on windows I see

error: [WinError 32] The process cannot access the file because it is being used by another process: 'build\\bdist.win-amd64\\wheel\\flashinfer\\data\\cutlass\\include\\cutlass\\experimental\\distributed\\device'

      [end of output]

Would you know what is going wrong?

bowang007 · 2025-04-21T21:35:49Z

@bowang007 on linux converter tests I see- FAILED automatic_plugin/test_flashinfer_rmsnorm.py::TestAutomaticPlugin::test_rmsnorm_float_0 - RuntimeError: FlashInfer requires sm75+ and on windows I see error: [WinError 32] The process cannot access the file because it is being used by another process: 'build\bdist.win-amd64\wheel\flashinfer\data\cutlass\include\cutlass\experimental\distributed\device'
  [end of output]
Would you know what is going wrong?

Hi @apbose ,

When you do the cross-compile, what is the sm version that you are compiling into?
If flashinfer library is not support on other platform, maybe we could just turn these tests off

apbose · 2025-04-21T21:48:47Z

Hmm @bowang007 are you suggesting the above wrt to the linux tests or the windows test? The error seems to be coming specifically in pytorch/TensorRT/tree/main/tests/py/dynamo/automatic_plugin/test_flashinfer_rmsnorm.py tests in linux
which is independent of cross compilation. The windows error I am not sure. Am I missing something?

bowang007 · 2025-04-21T21:56:55Z

wrt

Hi @apbose ,
I mean it is not because of cross-compilation, it is because of the flashinfer library here https://github.com/flashinfer-ai/flashinfer might just not support running on the platform with sm 75 or windows.

bowang007 · 2025-04-21T21:59:41Z

wrt

Hi @apbose , I mean it is not because of cross-compilation, it is because of the flashinfer library here https://github.com/flashinfer-ai/flashinfer might just not support running on the platform with sm 75 or windows.

@apbose can we do something like turning off these tests for other platforms for now?

facebook-github-bot added the cla signed label Apr 14, 2025

github-actions bot added component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Apr 14, 2025

github-actions bot requested a review from peri044 April 14, 2025 20:35

apbose marked this pull request as draft April 14, 2025 20:35

narendasan reviewed Apr 14, 2025

View reviewed changes

apbose marked this pull request as ready for review April 15, 2025 21:16

narendasan reviewed Apr 15, 2025

View reviewed changes

apbose commented Apr 15, 2025

View reviewed changes

apbose added the needs-release-cherrypick label Apr 15, 2025

github-actions bot requested a review from gs-olive April 15, 2025 21:50

github-actions bot added component: tests Issues re: Tests component: runtime labels Apr 18, 2025

apbose requested a review from narendasan April 18, 2025 23:26

narendasan reviewed Apr 18, 2025

View reviewed changes

apbose force-pushed the cross_compile_code_reorg_and_corr branch from 3f8ab4c to 2934660 Compare April 19, 2025 00:17

narendasan approved these changes Apr 19, 2025

View reviewed changes

apbose added 2 commits April 21, 2025 16:15

slight code reorg and bug correction for cross_compile

108b0ec

adding test case for multiple outputs and moving op to register_meta_ops

f8f0f55

apbose force-pushed the cross_compile_code_reorg_and_corr branch from 2934660 to f8f0f55 Compare April 21, 2025 23:15

apbose merged commit dc36709 into main Apr 22, 2025
65 of 68 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

slight code reorg and bug correction for cross_compile #3472

slight code reorg and bug correction for cross_compile #3472

apbose commented Apr 14, 2025

narendasan Apr 14, 2025

apbose Apr 15, 2025

narendasan Apr 15, 2025 •

edited

Loading

narendasan Apr 15, 2025

apbose Apr 15, 2025

narendasan Apr 18, 2025

narendasan left a comment

apbose commented Apr 21, 2025 •

edited

Loading

bowang007 commented Apr 21, 2025

apbose commented Apr 21, 2025

bowang007 commented Apr 21, 2025

bowang007 commented Apr 21, 2025

slight code reorg and bug correction for cross_compile #3472

slight code reorg and bug correction for cross_compile #3472

Conversation

apbose commented Apr 14, 2025

narendasan Apr 14, 2025

Choose a reason for hiding this comment

apbose Apr 15, 2025

Choose a reason for hiding this comment

narendasan Apr 15, 2025 • edited Loading

Choose a reason for hiding this comment

narendasan Apr 15, 2025

Choose a reason for hiding this comment

apbose Apr 15, 2025

Choose a reason for hiding this comment

narendasan Apr 18, 2025

Choose a reason for hiding this comment

narendasan left a comment

Choose a reason for hiding this comment

apbose commented Apr 21, 2025 • edited Loading

bowang007 commented Apr 21, 2025

apbose commented Apr 21, 2025

bowang007 commented Apr 21, 2025

bowang007 commented Apr 21, 2025

narendasan Apr 15, 2025 •

edited

Loading

apbose commented Apr 21, 2025 •

edited

Loading