Add array api compliance report workflow - #25
Conversation
|
@ev-br what should be included as skips / xfails? Most of our skips could just as well be xfails. I know they're there, but don't need to be reminded of them in every test run, and there's no benefit to them running. There's one skip that is really needed: with recent Besides that, should we skip/xfail anything else for the compliance testing? Before merging, I'll stop this report from being generated on every |
|
ATM compliance reporting has no intelligence about xfails, AFAIU, it only reports for failures and skips. For the array api test suite, skips are better for the CI run time, https://github.com/data-apis/array-api-tests#turning-xfails-into-skips Re frequency, AFAIU the dashboard side of things runs on a daily cron. Push to main + manual trigger sounds compatible to me, too. |
|
Well, the runtime is fast enough, and it's on our CI anyway. It sounds like this is fine, so I'll go ahead and merge. LMK if there's trouble with the report that needs to be fixed here. |
Closes #24. Registered at data-apis/array-api-tests-compliance-data#9.