Investigate test_py_3.12_backend failures for onnx==1.17.0

[ahsan-ca]

We are hitting test case failures for test_py_3.12_backend when the onnx version is set to 1.17.0.

Once the failures are fixed, the change in #3842 can be reverted.

To run the test case with verbose output: ctest -R test_py_3.12_backend . -V.

Machine: MI300

======================================================================
ERROR: test_convtranspose_group_2_cpu (__main__.OnnxBackendNodeModelTest.test_convtranspose_group_2_cpu)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 303, in device_test_func
    return test_func(*args, device, **merged_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 393, in run
    prepared_model = self.backend.prepare(model, device, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 126, in prepare
    return cls.prepare(bin, device, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 112, in prepare
    inf = migraphx.parse_onnx_buffer(model)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: /code/AMDMIGraphX/AMDMIGraphX/src/include/migraphx/op/convolution_backwards.hpp:85: compute_shape: CONVOLUTION_BACKWARDS: mismatched channel numbers

======================================================================
ERROR: test_convtranspose_group_2_image_3_cpu (__main__.OnnxBackendNodeModelTest.test_convtranspose_group_2_image_3_cpu)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 303, in device_test_func
    return test_func(*args, device, **merged_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 393, in run
    prepared_model = self.backend.prepare(model, device, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 126, in prepare
    return cls.prepare(bin, device, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 112, in prepare
    inf = migraphx.parse_onnx_buffer(model)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: /code/AMDMIGraphX/AMDMIGraphX/src/include/migraphx/op/convolution_backwards.hpp:85: compute_shape: CONVOLUTION_BACKWARDS: mismatched channel numbers

======================================================================
ERROR: test_reduce_max_empty_set_cpu (__main__.OnnxBackendNodeModelTest.test_reduce_max_empty_set_cpu)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 303, in device_test_func
    return test_func(*args, device, **merged_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 393, in run
    prepared_model = self.backend.prepare(model, device, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 126, in prepare
    return cls.prepare(bin, device, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 114, in prepare
    inf.to_py(), inf))
    ^^^^^^^^^^^
RuntimeError: /code/AMDMIGraphX/AMDMIGraphX/src/shape.cpp:367: lens: SHAPE: lens() called on a dynamic shape

======================================================================
ERROR: test_reduce_sum_empty_axes_input_noop_cpu (__main__.OnnxBackendNodeModelTest.test_reduce_sum_empty_axes_input_noop_cpu)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 303, in device_test_func
    return test_func(*args, device, **merged_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 393, in run
    prepared_model = self.backend.prepare(model, device, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 126, in prepare
    return cls.prepare(bin, device, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 114, in prepare
    inf.to_py(), inf))
    ^^^^^^^^^^^
RuntimeError: /code/AMDMIGraphX/AMDMIGraphX/src/shape.cpp:367: lens: SHAPE: lens() called on a dynamic shape

======================================================================
ERROR: test_resize_tf_crop_and_resize_extrapolation_value_cpu (__main__.OnnxBackendNodeModelTest.test_resize_tf_crop_and_resize_extrapolation_value_cpu)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 303, in device_test_func
    return test_func(*args, device, **merged_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 393, in run
    prepared_model = self.backend.prepare(model, device, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 126, in prepare
    return cls.prepare(bin, device, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 112, in prepare
    inf = migraphx.parse_onnx_buffer(model)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: /code/AMDMIGraphX/AMDMIGraphX/src/onnx/parse_resize.cpp:111: get_coord_trans_mode: PARSE_RESIZE: "tf_crop_and_resize" mode is not supported!

======================================================================
ERROR: test_resize_upsample_sizes_nearest_not_smaller_cpu (__main__.OnnxBackendNodeModelTest.test_resize_upsample_sizes_nearest_not_smaller_cpu)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 303, in device_test_func
    return test_func(*args, device, **merged_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/test/py/venv-onnx/3.12/lib/python3.12/site-packages/onnx/backend/test/runner/__init__.py", line 393, in run
    prepared_model = self.backend.prepare(model, device, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 126, in prepare
    return cls.prepare(bin, device, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/AMDMIGraphX/AMDMIGraphX/build/lib/onnx_migraphx/backend.py", line 112, in prepare
    inf = migraphx.parse_onnx_buffer(model)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: /code/AMDMIGraphX/AMDMIGraphX/src/include/migraphx/op/resize.hpp:199: compute_shape: RESIZE: size/scale input's size must match rank of input X

----------------------------------------------------------------------
Ran 2874 tests in 2366.805s

FAILED (errors=6, skipped=2115)
Default GPU device is used ....
3/3 Test #test_py_3.12_backend .............***Failed  2369.77 sec

The following tests passed:
        py_3.12_onnx_initialize_env
        py_3.12_onnx_setup_env

67% tests passed, 1 tests failed out of 3

Total Test time (real) = 2375.60 sec

The following tests FAILED:
        404 - test_py_3.12_backend (Failed)

[eddieliao]

Couple notes after investigating with Chris:

• All the failing tests were added in 1.17, none are regressions from existing tests
• Could likely exclude and bump back up to 1.17 if needed
  • Should go through tests and check if any of them we should be supporting
• Python tests are not built into binaries (in AMDMIGraphX/build/bin)
  • This means that they are not included in migraphx-tests and subsequently QA does not run them