Newer versions of manylinux

[johannesjh]

TLDR: I think we can bump the max version number of manylinux to whatever happens to be the max supported version in the operating system where req2flatpak is run.

In detail:

The problem, failing tests: I added additional testdata by running req2flatpak --platform-info > tests/cp311-x86_64.platforminfo.json and found that tests.test_platform_factory.RegressionTest.test fails now. The reason is that my python installation supports newer versions of manylinux, including 2.36 and 2.37, whereas req2flatpak's req2flatpak.PlatformFactory._cp3_linux_tags method only knows of manylinux up to version 2.35

Background research: According to https://github.com/pypa/manylinux and https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/, the (newer) manylinux versions follow PEP 600, which states that a wheel tagged manylinux_x_y shall work on any distro based on glibc>=x.y. (And indeed, running ldd --version confirms that my system comes with glibc 3.37).

There is really no way for req2flatpak.PlatformFactory._cp3_linux_tags to guess the glibc version that might be installed on a system. Because, e.g., an old system with an old version of glibc may nonetheless have a new version of python installed, and an old version of python might be installed on a new system; I don't think there is a strict correlation between glibc versions and python versions.

Implementation options, with regard to req2flatpak, the following options come to my mind, in order of increasing complexity:

• Option 1, done: We could leave things as is, namely: The maximum manylinux version currently is a hardcoded default in req2flatpak. We can occasionally bump the version number when situations like this ticket occur.
• Option 2, won't do: We could add a commandline option, allowing users to specify the maximum manylinux version. But so far, nobody has needed this option. => I'd therefore opt for "won't do".
• Option 3, done: We could allow users to specify a list of target platform tags that they want to target. In fact, this is possible already using the req2flatpak's python API, as documented in https://johannesjh.github.io/req2flatpak/main/api.html#specifying-target-platforms. => This is done already, and it somewhat eliminates the need for option (2).

So, which specific max version number should req2flatpak assume to be supported by a platform? I'd argue to assume a version that is widespread amongst stable linux distributions. It is better for req2flatpak to assume a low version of manylinux and download compatible packages than the other way round. The current debian stable (version 12, "bookworm") ships with glibc v2.36, i.e., manylinux 2.36. I'd say we should use this.

[cbm755]

I only skimmed this but do you mean min rather than max?

[johannesjh]

No, I did mean "max", but I can see how these things can be confusing.

Some more explanations, including references to the code.
req2flatpak, the way it is implemented now, does the following when we (as users) specify a target platform:

• req2flatpak's commandline api takes one (or multiple) target platform strings as input, e.g., 310-x86_64 for python 3.10 on x86 architecture
• req2flatpak.PlatformFactory.from_string generates a Platform object from this input. This object contains a list of platform tags that this platform (probably) supports. More specifically, the list contains tags for different versions of manylinux that the platform (probably) supports. Why "probably"? - since req2flatpak cannot know which specific versions of manylinux are supported on any given platform described using a simple string such as 310-x86_64, it has to do some guesswork.
• This guesswork is implemented in req2flatpak.PlatformFactory._cp3_linux_tags. It currently lists manylinux versions up to 2.37

...so this ticket asks if req2flatpak.PlatformFactory._cp3_linux_tags should take newer (= higher maximum) versions of manylinux into account (e.g., versions 2.36 and 2.37) when it heuristically guesses platform tags that a platform is likely to support.

[real-yfprojects]

Is there a reason for limiting the maximum manylinux version other than our current code architecture? In that case one could think about removing the need for a maximum version.

[johannesjh]

Thank you for discussing this topic together!

Is there a reason for limiting the maximum manylinux version other than our current code architecture? In that case one could think about removing the need for a maximum version.

I would agree that the current manylinux version provides a reasonable default value for the CLI.

Nonetheless, users may need to specify a different maximum manylinux version when cross-compiling. For example, a user may run req2flatpak on a recently updated operating system, working on an app which targets an older flatpak SDK and platform.

I'll try to sketch a solution:

• Add flexibility to the inner workings of req2flatpak: Todo, the python methods _cp3_linux_tags and from_python_version_and_arch shall accept an additional argument, allowing to set a custom maximum manylinx version instead of the hardcoded value.
• Keep the CLI simple: Todo, the commandline API uses a helper function to determine the the current manylinux version and use this version as default value.
• (Keep the python API flexible: No additional changes needed, it is flexible already).

[real-yfprojects]

Sounds reasonable!