Exclude failing tests for ONNX 1.17.0

[eddieliao]

Reverts #3842 and excludes tests that are failing in ONNX 1.17.0. Should revisit at some time to assess whether we should be supporting them, but this allows us to upgrade to a newer version of ONNX for the time being.

Closes #3843.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.35%. Comparing base (b19d136) to head (8953f9e).
Report is 5 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3846   +/-   ##
========================================
  Coverage    92.35%   92.35%           
========================================
  Files          519      519           
  Lines        22311    22311           
========================================
  Hits         20606    20606           
  Misses        1705     1705           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[migraphx-bot]

Test	Batch	Rate new 8953f9	Rate old 105928	Diff	Compare
torchvision-resnet50	64	3,236.54	3,237.56	-0.03%	✅
torchvision-resnet50_fp16	64	6,882.93	6,876.50	0.09%	✅
torchvision-densenet121	32	2,439.51	2,438.24	0.05%	✅
torchvision-densenet121_fp16	32	4,188.61	4,204.55	-0.38%	✅
torchvision-inceptionv3	32	1,614.66	1,615.26	-0.04%	✅
torchvision-inceptionv3_fp16	32	2,679.95	2,682.88	-0.11%	✅
cadene-inceptionv4	16	751.30	750.23	0.14%	✅
cadene-resnext64x4	16	810.55	810.22	0.04%	✅
slim-mobilenet	64	6,659.87	6,659.63	0.00%	✅
slim-nasnetalarge	64	196.90	196.82	0.04%	✅
slim-resnet50v2	64	3,430.33	3,429.65	0.02%	✅
bert-mrpc-onnx	8	1,143.15	1,143.03	0.01%	✅
bert-mrpc-tf	1	482.66	489.79	-1.45%	✅
pytorch-examples-wlang-gru	1	477.47	477.72	-0.05%	✅
pytorch-examples-wlang-lstm	1	447.00	442.18	1.09%	✅
torchvision-resnet50_1	1	802.72	804.65	-0.24%	✅
cadene-dpn92_1	1	431.45	429.49	0.46%	✅
cadene-resnext101_1	1	392.86	391.98	0.23%	✅
onnx-taau-downsample	1	371.25	370.23	0.27%	✅
dlrm-criteoterabyte	1	31.80	31.83	-0.11%	✅
dlrm-criteoterabyte_fp16	1	51.06	51.06	0.00%	✅
agentmodel	1	8,613.86	8,648.27	-0.40%	✅
unet_fp16	2	58.29	58.28	0.02%	✅
resnet50v1_fp16	1	1,025.93	1,037.98	-1.16%	✅
resnet50v1_int8	1	796.86	804.65	-0.97%	✅
bert_base_cased_fp16	64	1,164.31	1,165.40	-0.09%	✅
bert_large_uncased_fp16	32	361.58	361.67	-0.02%	✅
bert_large_fp16	1	200.20	200.43	-0.11%	✅
distilgpt2_fp16	16	2,219.05	2,216.48	0.12%	✅
yolov5s	1	530.22	530.12	0.02%	✅
tinyllama	1	43.60	43.65	-0.11%	✅
vicuna-fastchat	1	43.89	43.80	0.22%	✅
whisper-tiny-encoder	1	412.13	412.43	-0.07%	✅
whisper-tiny-decoder	1	412.23	410.80	0.35%	✅
yolov10	1	nan	nan	nan%	❌
llama2_7b	1	nan	nan	nan%	❌
qwen1.5-7b	1	nan	nan	nan%	❌
phi3-3.8b	1	nan	nan	nan%	❌
mask-rcnn	1	nan	nan	nan%	❌
llama3-8b	1	nan	nan	nan%	❌
whisper-large-encoder	1	nan	nan	nan%	❌
whisper-large-decoder	1	nan	nan	nan%	❌
mistral-7b	1	nan	nan	nan%	❌
FLUX.1-schnell	1	nan	nan	nan%	❌

This build is not recommended to merge 🔴

[migraphx-bot]

     ✅ bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

     ✅ bert-mrpc-tf: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

     ✅ torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

     ✅ cadene-dpn92_1: PASSED: MIGraphX meets tolerance

     ✅ cadene-resnext101_1: PASSED: MIGraphX meets tolerance

     ✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

     ✅ agentmodel: PASSED: MIGraphX meets tolerance

     ✅ unet: PASSED: MIGraphX meets tolerance

     ✅ resnet50v1: PASSED: MIGraphX meets tolerance

     ✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

     ✅ bert_large: PASSED: MIGraphX meets tolerance

     ✅ yolov5s: PASSED: MIGraphX meets tolerance

     ✅ tinyllama: PASSED: MIGraphX meets tolerance

     ✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

     ✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

❌llama2_7b: ERROR - check error output

usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask 1 256 @attention_mask 1 256

❌qwen1.5-7b: ERROR - check error output

usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask position_ids 1 256 @attention_mask 1 256 @position_ids 1 256

❌phi3-3.8b: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/phi3-3.8b/model.onnx

❌mask-rcnn: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/mask-rcnn/MaskRCNN-10.onnx

❌llama3-8b: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/llama3-8b/model.onnx

❌whisper-large-encoder: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/whisper-large/encoder_model.onnx

❌whisper-large-decoder: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/whisper-large/decoder_model.onnx

❌mistral-7b: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/mistral-7b/model.onnx

❌FLUX.1-schnell: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/FLUX.1-schnell/text_encoder/model.onnx

[ahsan-ca]

Thanks for doing this. Looks good.

Should revisit at some time to assess whether we should be supporting them.

It would be good to open an issue, for tracking purposes.

[eddieliao]

Opened an issue to track assessing each test for support: #3849