Test pip editable install with meson

[user202729]

Tests pip editable install with meson on GitHub Actions.

This uncovers several failing tests, which are fixed by (see dependencies below)

Note: review is hard because of the large number of dependencies. It is probably easier to look only in .github folder.

📝 Checklist

• The title is concise and informative.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.

⌛ Dependencies

• Use import_module instead of find_spec #39423
• Apply sort and filter of walk_packages consistently #39498
• Fix more doctests in meson_editable install #39494
• Fix bug with sage_getfile in meson editable install #39424
• Specify use_sources=True in cython compilation #39275
• Improve sage_getfile by looking at __init__ #39499

[github-actions]

Documentation preview for this PR (built with commit 43142c7; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[tobiasdiez]

I'm fine with this. @kwankyu any opinions?

[kwankyu]

No.

[user202729]

We may also do something like "only test Python 3.11 non-editable & Python 3.12 editable" to save some carbon. Thoughts? (the old Conda matrix does something similar.)

[tobiasdiez]

We may also do something like "only test Python 3.11 non-editable & Python 3.12 editable" to save some carbon. Thoughts? (the old Conda matrix does something similar.)

I like this! Having only one "editable" run should be sufficient, right? (github workflows have the include or so keyword to add a single config to a matrix)

[user202729]

Okay done (use the trick first used in https://github.com/sagemath/sage/pull/39369/files )

[tobiasdiez]

This now makes it hard to extend the matrix by other OS for PRs (where you would like to run all non-editable versions for all pythons and all systems).

Does the following work?

editable: 
- false
- ${{ github.event_name != 'pull_request' }} # PR: false (which is then ignored), Push: true

include: 
          - os: ubuntu
            python: 3.12
            editable: true # One additional editable run for PRs

[user202729]

I don't understand what combination you are trying to do. The way it currently works is

in PR, only run the combinations listed below; otherwise, run all combinations

As long as in non-PR you want to run all combinations you can just edit the matrix. Hopefully you don't need to run too many combinations in pull request that you can afford listing each configuration explicitly.

So for example if you were to add windows and mac to the os: [ubuntu], then you could modify the list below to e.g.

        include:
          - os: ubuntu
            python: 3.11
            editable: false
          - os: mac
            python: 3.12
            editable: false
          - os: windows
            python: 3.12
            editable: true

or something like that. (which is probably sufficient right? Each factor got tested at least once)

---

In your suggestion, on push all combinations will be run, and on pull request 3 combinations will be run (which also work, I think).

[tobiasdiez]

Hopefully you don't need to run too many combinations in pull request that you can afford listing each configuration explicitly.

That's the problem. Once we add macos and windows to the matrix, it will run 3 (os) x 3 (python) systems for each PR (with proper caching this hopefully isn't too much total CI time). I prefer not to specify all of these combinations explicitly.

[user202729]

I think one problem is do you really want to run 10 per PR. It seems like 3 is enough (see my suggestion above).

Although… I think if it gets sufficiently complicated it might be better to use full-blown code to generate the include list. (Does whatever language that GitHub Actions use have a for loop or list comprehension or map etc.? Seems not.)

e.g. https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow#example-using-an-output-to-define-two-matrices could be used. But then this implies we need a separate script written in something like Python (is this preinstalled?) to generate the JSON.

Is it worth it or is this overengineering?

[tobiasdiez]

Is it worth it or is this overengineering?

It's a tad too complicated in my opinion.

I think one problem is do you really want to run 10 per PR. It seems like 3 is enough

In my opinion, issues should be caught at the PR level. And the only way to observe a bug, say on MacOS with Python 3.11, is to run exactly that combination. Preferably, it would run only once for the merge and not at every push to a PR but that's not possible with our current setup.

But anyway, for now I would prefer a solution that adds exactly one "editable" run for PRs (and perhaps for all Python versions on push to develop).

[tobiasdiez]

Thanks again for your fixes of the editable mode!