Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wxPython fails to build on free threaded python 3.13t due to __pyx_vectorcallfunc related error in nanosvg-extension (from source, no wheel available) #2707

Open
florianber1 opened this issue Feb 20, 2025 · 8 comments

Comments

@florianber1
Copy link

Installing wxPython with pip for python 3.13t fails during generation of the nanosvg extension. Pip build from source because no wheel may be available.
My system: Windows 10 Home 64-bit (10.0, Build 1945)
Python: experimental free threading version Python 3.13t
Environment: created venv, activated manually by navigating to \Scripts and execute batch activate.bat - then stepped out on venv root directory

Packages installed before wxPython (with pip install -U): setuptools, wheel, requests, six, attrdict3

Package Version


attrdict3 2.0.2
certifi 2025.1.31
charset-normalizer 3.4.1
idna 3.10
pip 25.0.1
requests 2.32.3
setuptools 75.8.0
six 1.17.0
urllib3 2.3.0
wheel 0.45.1

The failure happens because __pyx_vectorcallfunc is not recognized in CYTHON_METH_FASTCALL function declarations/definitions. For example - line 25916 in file wx\svg\nanosvg:

#if CYTHON_METH_FASTCALL

static PyObject *__Pyx_PyVectorcall_FastCallDict_kw(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw)
{
    PyObject *res = NULL;
    PyObject *kwnames;
    PyObject **newargs;
    ...

It seems to be recognized within function scope (no error in compiler output, see below), for example: line 23633 to 23649

...
if (kwargs == NULL) {

        #if CYTHON_VECTORCALL
        #if PY_VERSION_HEX < 0x03090000
        vectorcallfunc f = _PyVectorcall_Function(func);
        #else
        vectorcallfunc f = PyVectorcall_Function(func);
        #endif
        if (f) {
            return f(func, args, (size_t)nargs, NULL);
        }
        #elif defined(__Pyx_CyFunction_USED) && CYTHON_BACKPORT_VECTORCALL
        if (__Pyx_CyFunction_CheckExact(func)) {
            __pyx_vectorcallfunc f = __Pyx_CyFunction_func_vectorcall(func);
            if (f) return f(func, args, (size_t)nargs, NULL);
        }
        #endif
    }
...

I have tried the following without success:

  • Installed the packages from the list devel.txt and install.txt in \requirements
  • Installed the latest Cython (3.0.12) instead of the required version 3.0.10.
  • Installed the latest Cython by cloning from the gitmaster (3.1.xx alpha)

The issue must be buried in the free threading architecture - wxPython installs and/or builds from 3.13 successfully.

The relevant compiler error is posted below. I also attached the full redirected stdout as log file.

     D:\Python\wxPythonSrcBuild3.13t\Scripts\python.exe setup-wxsvg.py build_ext --inplace
      running build_ext
      building 'wx.svg._nanosvg' extension
      creating build\wxsvg\temp.win-amd64-cpython-313t\Release\wx\svg
      "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DNANOSVG_IMPLEMENTATION=1 -DNANOSVGRAST_IMPLEMENTATION=1 -DNANOSVG_ALL_COLOR_KEYWORDS=1 -DPy_GIL_DISABLED=1 -Iext/nanosvg/src -ID:\Python\wxPythonSrcBuild3.13t\include "-IC:\Program Files\Python\include" "-IC:\Program Files\Python\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\Include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" /Tcwx/svg\_nanosvg.c /Fobuild/wxsvg\temp.win-amd64-cpython-313t\Release\wx\svg\_nanosvg.obj
      _nanosvg.c
      ext/nanosvg/src\nanosvg.h(1450): warning C4244: 'initializing': conversion from 'double' to 'float', possible loss of data
      ext/nanosvg/src\nanosvg.h(1458): warning C4244: 'initializing': conversion from 'double' to 'float', possible loss of data
      ext/nanosvg/src\nanosvg.h(1491): warning C4244: '=': conversion from 'double' to 'float', possible loss of data
      ext/nanosvg/src\nanosvg.h(1770): warning C4244: '=': conversion from 'double' to 'float', possible loss of data
      ext/nanosvg/src\nanosvg.h(1791): warning C4244: '=': conversion from 'double' to 'float', possible loss of data
      ext/nanosvg/src\nanosvg.h(2553): warning C4244: '=': conversion from 'double' to 'float', possible loss of data
      ext/nanosvg/src\nanosvg.h(2557): warning C4244: '=': conversion from 'double' to 'float', possible loss of data
      ext/nanosvg/src\nanosvg.h(2561): warning C4244: '=': conversion from 'double' to 'float', possible loss of data
      ext/nanosvg/src\nanosvg.h(2565): warning C4244: '=': conversion from 'double' to 'float', possible loss of data
      wx/svg\_nanosvg.c(2455): error C2146: syntax error: missing ')' before identifier 'vc'
      wx/svg\_nanosvg.c(2455): error C2081: '__pyx_vectorcallfunc': name in formal parameter list illegal
      wx/svg\_nanosvg.c(2455): error C2061: syntax error: identifier 'vc'
      wx/svg\_nanosvg.c(2455): error C2059: syntax error: ';'
      wx/svg\_nanosvg.c(2455): error C2059: syntax error: ','
      wx/svg\_nanosvg.c(2455): error C2059: syntax error: ')'
      wx/svg\_nanosvg.c(25912): error C2146: syntax error: missing ')' before identifier 'vc'
      wx/svg\_nanosvg.c(25912): error C2081: '__pyx_vectorcallfunc': name in formal parameter list illegal
      wx/svg\_nanosvg.c(25912): error C2061: syntax error: identifier 'vc'
      wx/svg\_nanosvg.c(25912): error C2059: syntax error: ';'
      wx/svg\_nanosvg.c(25912): error C2059: syntax error: ','
      wx/svg\_nanosvg.c(25912): error C2059: syntax error: ')'
      wx/svg\_nanosvg.c(25957): error C2146: syntax error: missing ')' before identifier 'vc'
      wx/svg\_nanosvg.c(25957): error C2081: '__pyx_vectorcallfunc': name in formal parameter list illegal
      wx/svg\_nanosvg.c(25957): error C2061: syntax error: identifier 'vc'
      wx/svg\_nanosvg.c(25957): error C2059: syntax error: ';'
      wx/svg\_nanosvg.c(25957): error C2059: syntax error: ','
      wx/svg\_nanosvg.c(25957): error C2059: syntax error: ')'
      wx/svg\_nanosvg.c(26646): error C2065: '__pyx_vectorcallfunc': undeclared identifier
      wx/svg\_nanosvg.c(26646): error C2146: syntax error: missing ';' before identifier 'vc'
      wx/svg\_nanosvg.c(26646): error C2065: 'vc': undeclared identifier
      wx/svg\_nanosvg.c(26646): warning C4047: '=': 'int' differs in levels of indirection from 'vectorcallfunc'
      wx/svg\_nanosvg.c(26647): error C2065: 'vc': undeclared identifier
      wx/svg\_nanosvg.c(26649): warning C4013: '__Pyx_PyVectorcall_FastCallDict' undefined; assuming extern returning int
      wx/svg\_nanosvg.c(26649): error C2065: 'vc': undeclared identifier
      wx/svg\_nanosvg.c(26649): warning C4047: 'return': 'PyObject *' differs in levels of indirection from 'int'
      error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\cl.exe' failed with exit code 2
      Command 'D:\Python\wxPythonSrcBuild3.13t\Scripts\python.exe setup-wxsvg.py build_ext --inplace' failed with exit code 1.
      Finished command: build_others (0.742s)
      Finished command: build_py (7m42.396s)
      Finished command: build (13m9.458s)
      Command '"D:\Python\wxPythonSrcBuild3.13t\Scripts\python.exe" -u build.py build' failed with exit code 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for wxPython
  Running setup.py clean for wxPython
Failed to build wxPython
ERROR: Failed to build installable wheels for some pyproject.toml based projects (wxPython)
@echoix
Copy link
Contributor

echoix commented Feb 20, 2025

It might (would) be solved if the .c file wasn't committed, and let a modern build backend build from source, as the pyx file would be used. Committing a .c isn't recommended by cython now with the python build improvements, where not having cython already installed isn't an issue anymore, as it can be specified as a build-time requirement in pyproject.toml. All that is assuming the issue is already solved in a recent Cython release.
For your build, maybe removing the requirement for the cython to not be language_level=2, but language_level=3str or language_level=3 might help

@florianber1
Copy link
Author

For your build, maybe removing the requirement for the cython to not be language_level=2, but language_level=3str or language_level=3 might help

I am not very familiar with cython and building from source. Where do I need to specifiy language_level=3 in the files or command line?

@echoix
Copy link
Contributor

echoix commented Feb 21, 2025

For cython version:

Phoenix/pyproject.toml

Lines 72 to 78 in 5a13d13

[build-system]
requires = [
"setuptools>=70.1",
"cython == 3.0.10",
"requests >= 2.26.0",
"sip == 6.10.0",
]

For the Cython config:

Phoenix/setup-wxsvg.py

Lines 50 to 54 in 5a13d13

if have_cython:
modules = cythonize([module],
compiler_directives={'embedsignature': True,
'language_level':2,
})

File to delete to make sure it gets recreated:

https://github.com/wxWidgets/Phoenix/blob/5a13d13c316c0e396355d648ec910e9c3934886d/wx/svg/_nanosvg.c

Since you're on Windows, the CI workflows can build you a wheel.
Doing this on a branch on your fork, and doing a PR against your fork (not here), you'd be able to download the artifact. If it's a new fork, make sure to enable actions on your forked repo first.

You'd also need to add the free threaded version to the Python versions to build wheels for:

Add "3.13t" to

python-version: [ '3.9', '3.10', '3.11', '3.12', '3.13' ]

With these changes, you'd get a new wheel in about 45 minutes

@florianber1
Copy link
Author

Took me a while but I managed to create a branch with the modified files and deleted _nanosvg.c, with that branch I can now do a PR against the fork as you said.
Just to be sure: If I do a PR, will the CI workflow ci-build only build the wheels and NOT do the other things (publish to pyPI, create a github release, and upload a snapshot build)? In my understanding, it should not as do not want to push a tag to the repo - and it should not do that automatically, right?

@echoix
Copy link
Contributor

echoix commented Feb 21, 2025

Took me a while but I managed to create a branch with the modified files and deleted _nanosvg.c, with that branch I can now do a PR against the fork as you said.

Great!

Just to be sure: If I do a PR, will the CI workflow ci-build only build the wheels and NOT do the other things (publish to pyPI, create a github release, and upload a snapshot build)? In my understanding, it should not as do not want to push a tag to the repo - and it should not do that automatically, right?

In a PR, just like here, it won't happen. Unless you specifically added some pypi authentication token that happen to be valid to push to wxPython (which only like two people are allowed), the upload step would fail when merging to your fork. It fails on mine.

It also won't create a release, on your fork, unless you manually do so (not by actions). It also can't make a new file to the wxPython downloads (snapshot), unless you have added secrets called "RIOBU_SSH_KEY" and "RIOBU_KNOWN_HOSTS" that are valid for that upload server, which most probably would never be true.

You can take a look at my fork to make sure.

Come back if you need more help, or tag me in your PR so I can take a look at the workflow. Depending on your results afterwards, if you have a good experience with the results, we might consider adding support for it here too. (It could be your PR :) )

@florianber1
Copy link
Author

florianber1 commented Feb 22, 2025

I tried a few things, and discovered some likely blocks (for the workflow):

  1. (All) action.python-setup does not provide python-3.13t. Has to be fixed by the people from the related repo.
  2. (Ubuntu) fails building the cryptography module. The error is unspecific (build cryptography-ciff failed), but the warnings suggest that it has to do with the pyLIMITED API (cython). Has to be fixed by the people from the related repo.

Comments to the blocks

Ad 1)
Workaround: I found that using action.deadsnakes as a fallback in the workflow allows me to install python 3.13t - but only on Linux.
Shortcoming: No solution for windows. I was able to brute-build the wheel "manually" using pip and the (downloaded) tarball but the interpreter crashes when importing the installed package.
Solution: Wait for action.setup-python to confirm the PR of their new free-threaded branch that adds a new keyword "freethreading" and will allow the usage of 3.13t.

Ad 2)
Workaround None. I started to build-install cryptography in an isolated venv to try getting it built anyhow. Stopped after running into unspecific errors when the Rust compiler toolchain was unable to find the OpenSSL libraries although it could find the OPENSSL_PATH... I probably went a bit too deep with thisone.
Shortcoming: No solutions. Blocks build process on Linux.
Solution: Wait for the people from the cryptography to roll out a wheel for 3.13t. Several open issues are found on their repo, but those will not be solved soon.

Manual build

Although I should have known better, I tried to build manually again - with the instructions directly from the repo (Building-wxWidgets. I ran in an venv again, installed setuptools, wheel, six. I run into issue with build.py dox, because doxygen was unable to find the doxygen folder in the root-git. When I just created one, it was unable to find the next folder \ext\wxPython\docs\doxygen although it was clearly there (I ran submodule ...). Skipping to the next command build.py etg --nodoc did not help as well.

@echoix
Copy link
Contributor

echoix commented Feb 22, 2025

You're totally right, and I'm as surprised as you that setup-python doesn't support free-threaded builds. I quickly did a spot-check before writing my original answer, but as I receive the release notifications of the python-versions repo and I remembered seeing that they had free-threaded builds, and that all these artifacts are available for setup-python, I was surprised that the version file doesn't include them.

For a local build, I can suggest to not even try to create the source distribution. Download a recent artifact of an "sdist" (source distribution), that are the same across OSes, and try to build that, like: python -m pip install "./downloads/SomePackage-1.0.4.tar.gz" (example 8 of https://pip.pypa.io/en/stable/cli/pip_install/#examples)
Why I say recent, to have the latest changes in the build process from #2687 and later. Oh, and if some source changes are needed on your side, then the sdist from your PR on your fork should be required.

About some of your issues locally:

  • Installing setuptools directly shouldn't be needed once you have an sdist, it should be installed as a build dependency with recent build tools. Same thing for wheel, and six (I think it is now totally removed with PRs like Remove Python 2 compatibility and use of six module #2688).
  • Cloning: Did you do a recursive clone? It is not only the submodules of the repo to clone, but also the submodules of our submodules. git clone --recursive https://github.com/wxWidgets/Phoenix.git, or if already cloned, a foolproof command: git submodule update --init --recursive

@florianber1
Copy link
Author

florianber1 commented Feb 23, 2025

Well, it should not take very long, as the branch associates appear to be motivated, and merging requires just a few confirmations.

For my first manual build I followed the linux build instruction: install setuptools, six, wheel, download and install the wxPython tarball with pip. I skipped the linux packages (python-dev, etc.), and installed cython 3.1.a from repo with the pip install git+https://... . The wheel did build and pip installed the package. Sadly, the interpreter crashed right at the import python statement.

I did not clone the phoenix git recursively, but did so with the submodules. What, if I add --recursive every time? I may try it out. On the other hand, I might have more luck with downloading and building from the sdist artifacts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants