Skip to content

Add array api compliance report workflow - #25

Merged
mdhaber merged 6 commits into
mainfrom
array-api-tests-compliance
Sep 5, 2026
Merged

mdhaber merged 6 commits into
mainfrom
array-api-tests-compliance

Conversation

@mdhaber

@mdhaber mdhaber commented Sep 4, 2026 •

Copy link
Copy Markdown
Owner

@mdhaber

mdhaber commented Sep 4, 2026 •

Copy link
Copy Markdown
Owner Author

@ev-br what should be included as skips / xfails?

Most of our skips could just as well be xfails. I know they're there, but don't need to be reminded of them in every test run, and there's no benefit to them running.
https://github.com/mdhaber/mparray/blob/main/tools/xp-tests-skips.txt
The huge majority are in test_special_cases. mpmath calculations just don't implement those as the standard requires, and I haven't fixed them all yet. I'm going to propose fixes upstream.

There's one skip that is really needed: with recent hypothesis, array_api_tests/test_operators_and_elementwise_functions.py::test_pow is hanging. I imagine it is generating some example in which mpmath just takes a very long time. So we need to skip that.

Besides that, should we skip/xfail anything else for the compliance testing?

Before merging, I'll stop this report from being generated on every pull_request. I'll leave push to main and workflow dispatch, if that sounds good. I don't think we need a cron job unless you prefer, since pushes to main will probably be less frequent after this initial flurry of activity.

@ev-br

ev-br commented Sep 5, 2026

Copy link
Copy Markdown

ATM compliance reporting has no intelligence about xfails, AFAIU, it only reports for failures and skips.
There's some discussion at data-apis/array-api-tests-compliance-data#7

For the array api test suite, skips are better for the CI run time, https://github.com/data-apis/array-api-tests#turning-xfails-into-skips

Re frequency, AFAIU the dashboard side of things runs on a daily cron. Push to main + manual trigger sounds compatible to me, too.

@mdhaber

mdhaber commented Sep 5, 2026 •

Copy link
Copy Markdown
Owner Author

Well, the runtime is fast enough, and it's on our CI anyway. It sounds like this is fine, so I'll go ahead and merge. LMK if there's trouble with the report that needs to be fixed here.

Comment thread .github/workflows/report-compliance.yml Outdated
@mdhaber
mdhaber merged commit d205193 into main Sep 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider adding the Array API compatibility reporting

2 participants