support projects with pep517 deps

[graingert]

twisted-iocpsupport has a pep517 build-dep of "Cython":

https://github.com/twisted/twisted-iocpsupport/blob/f0b581834576ff02f3a4f4d17e76aa0740a2e911/pyproject.toml#L5

when I run fullrelease without the build-deps installed I get an error:

$ git clone [email protected]:twisted/twisted-iocpsupport.git
$ pipx run --spec="zest.releaser[recommended]" bumpversion   
Checking version bump for normal release.
CRITICAL: The setup.py of this package has an error:
Traceback (most recent call last):
CRITICAL: No version found.

If I manually install the deps I get this:

$ pipx run --pip-args="cython" --spec="zest.releaser[recommended]" bumpversion
Checking version bump for normal release.                 
Last tag: v1.0.0
Current version: 1.0.1.dev0
No version bump needed.

I think zest.releaser should be using pep517.meta.load(".").version to get the version

[mauritsvanrees]

I wonder if it would help if zest.releaser checks setup.cfg for a [metadata] version setting. That would mean we can get the version without needing to run python setup.py --version. That should fix the bumpversion command.

But when making a release, we still need to run python setup.py sdist, which would fail anyway:

$ python3.8 setup.py sdist
Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'

Your code does not work for me though:

>>> import pep517
>>> pep517.meta.load(".").version
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'pep517' has no attribute 'meta'
>>> pep517.__version__
'0.9.1'

And when I try the first example from the PyPI page I get an error, because the calling code is still responsible for installing the required dependencies:

>>> print(hooks.get_requires_for_build_wheel(config_options))
Traceback (most recent call last):
  File "/Users/maurits/tmp/twisted-iocpsupport/lib/python3.8/site-packages/pep517/_in_process.py", line 280, in <module>
    main()
  File "/Users/maurits/tmp/twisted-iocpsupport/lib/python3.8/site-packages/pep517/_in_process.py", line 263, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/Users/maurits/tmp/twisted-iocpsupport/lib/python3.8/site-packages/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
    return hook(config_settings)
  File "/Users/maurits/tmp/twisted-iocpsupport/lib/python3.8/site-packages/setuptools/build_meta.py", line 147, in get_requires_for_build_wheel
    return self._get_build_requires(
  File "/Users/maurits/tmp/twisted-iocpsupport/lib/python3.8/site-packages/setuptools/build_meta.py", line 128, in _get_build_requires
    self.run_setup()
  File "/Users/maurits/tmp/twisted-iocpsupport/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 7, in <module>
    from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'

And when I try the (deprecated) higher-level functions which install the build dependencies into a temporary environment and build a wheel/sdist using them, from the second example, I get:

>>> whl_filename = build_wheel(src, destination)
WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
You should consider upgrading via the '/Users/maurits/tmp/twisted-iocpsupport/bin/python -m pip install --upgrade pip' command.
Compiling twisted_iocpsupport/iocpsupport.pyx because it changed.
[1/1] Cythonizing twisted_iocpsupport/iocpsupport.pyx
/var/folders/26/1plvhxbs6yx7g_82v2xdxc500000gn/T/pep517-build-env-5t9ofqg2/lib/python3.8/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /Users/maurits/tmp/twisted-iocpsupport/twisted_iocpsupport/iocpsupport.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
running egg_info
creating twisted_iocpsupport.egg-info
writing twisted_iocpsupport.egg-info/PKG-INFO
writing dependency_links to twisted_iocpsupport.egg-info/dependency_links.txt
writing top-level names to twisted_iocpsupport.egg-info/top_level.txt
writing manifest file 'twisted_iocpsupport.egg-info/SOURCES.txt'
reading manifest file 'twisted_iocpsupport.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'twisted_iocpsupport.egg-info/SOURCES.txt'
WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
You should consider upgrading via the '/Users/maurits/tmp/twisted-iocpsupport/bin/python -m pip install --upgrade pip' command.
running bdist_wheel
running build
running build_ext
building 'twisted_iocpsupport.iocpsupport' extension
creating build
creating build/temp.macosx-10.15-x86_64-3.8
creating build/temp.macosx-10.15-x86_64-3.8/twisted_iocpsupport
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/usr/local/include -Itwisted_iocpsupport -I/Users/maurits/tmp/twisted-iocpsupport/include -I/Users/maurits/.pyenv/versions/3.8.5/include/python3.8 -c twisted_iocpsupport/iocpsupport.c -o build/temp.macosx-10.15-x86_64-3.8/twisted_iocpsupport/iocpsupport.o
twisted_iocpsupport/iocpsupport.c:631:10: fatal error: 'io.h' file not found
#include "io.h"

In other words: yes, the pep518 package is interesting and may be needed, or perhaps the build package builder that is intended to replace the deprecated functions. But it does not look like this will always work.
Tricky stuff.

[graingert]

@mauritsvanrees pep517.meta.load works for me:

$ git clone [email protected]:twisted/twisted-iocpsupport.git
$ pipx run --spec="pep517" python
⚠️  python is already on your PATH and installed at /usr/bin/python. Downloading and running anyway.
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pep517.meta
>>> pep517.meta.load(".").version
'1.0.1.dev0'

[graingert]

>>> whl_filename = build_wheel(src, destination)

build_wheel only works on windows, you just need the metadata

[graingert]

I wonder if it would help if zest.releaser checks setup.cfg for a [metadata] version setting.

the setup.cfg metadata version is an attr: field and could rely on build system requirements. Also the build system could consume the setup.cfg config in an arbitrary way as long as it fulfils pep517

[mauritsvanrees]

Ah, pep517.meta needs to be imported explicitly:

>>> import pep517
>>> pep517.meta.load(".").version
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'pep517' has no attribute 'meta'
>>> import pep517.meta
>>> pep517.meta.load(".").version
'1.0.1.dev0'

[mauritsvanrees]

From looking at https://zestreleaser.readthedocs.io/en/latest/versions.html, the seems zest.releaser can already read the metadata version from setup.cfg. We probably first try setup.py, and if this fails hard, we never reach the part of our code that reads setup.cfg.

If the version in setup.cfg is some dynamically calculated version, then zest.releaser might be able to read it, but will not be able to update this version.

[graingert]

zest.releaser bumpversion works here though (with cython installed)

[mauritsvanrees]

It's almost a year later, and nothing has been done by us in this area, sorry about that. And I have enough non-zest-releaser work on my plate to think that I will not do anything about this on short term. :-/

I read https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html today, which says all invocations of python setup.py are either broken or they only work by chance, and they should be replaced.

Specifically, python setup.py sdist could be replaced by calling build.

For the packages that I use it for, which are all pure Python, current zest.releaser still works fine. But we need to do some modernization before things really start falling apart.

If I still have some energy left after next week's Plone Conference, I could maybe work on this during the sprint.

[graingert]

pep517.meta:load has been succeeded by build.util:project_wheel_metadata

https://pypa-build.readthedocs.io/en/latest/api.html#build.util.project_wheel_metadata

[mauritsvanrees]

I have released a much cleaned up 7.0.0a1. Python 3 only, git only, less ancient workarounds.

I might pick up this or other related issues at some point, to bring us more in line with current Python packaging best practices, but I kinda hope that the cleanup makes it more inviting for others to contribute. :-)

[mauritsvanrees]

I have released 9.0.0a1. This uses build now to build packages, and can read pyproject.toml. That last part is partially already the case in 8.0.0.
So you could try one of those versions to see if that solves your problem.