Isolate macOS wheel builds from Homebrew #8497
Conversation
radarhere
changed the title
|
|
||
| # Ensure pkg-confg and cmake are available | ||
| build_pkg_config | ||
| python -m pip install cmake |
There was a problem hiding this comment.
| python -m pip install cmake | |
| python3 -m pip install cmake |
.github/workflows/wheels-test.sh
Outdated
| export PKG_CONFIG_PATH="/usr/local/opt/openblas/lib/pkgconfig" | ||
| if [ -f /opt/homebrew/lib/libfribidi.dylib ]; then | ||
| sudo cp /opt/homebrew/lib/libfribidi.dylib /usr/local/lib | ||
| # For Unix, Ensure fibidi is installed by the system. |
There was a problem hiding this comment.
| # For Unix, Ensure fibidi is installed by the system. | |
| # For Unix, ensure fribidi is installed by the system. |
| @@ -23,6 +23,8 @@ OPENJPEG_VERSION=2.5.2 | |||
| XZ_VERSION=5.6.3 | |||
| TIFF_VERSION=4.6.0 | |||
| LCMS2_VERSION=2.16 | |||
| RAQM_VERSION=0.7.1 | |||
There was a problem hiding this comment.
| RAQM_VERSION=0.7.1 | |
| RAQM_VERSION=0.10.2 |
There was a problem hiding this comment.
It definitely makes sense to use the most recent available version; but pillow-depends-main doesn't include this version. What's the procedure for updating that repo?
(Related: pkg-config, libXdmcp, and an updated webp sources should probably be added to that project)
There was a problem hiding this comment.
The theory is that pillow-depends-main is just a more reliable source for the dependencies, and PRs should work without it.
Once this is merged, I will likely add them there.
|
The manylinux failures look like they'll be fixed by this PR to multi build. The x86_64 macOS failures are probably a leakage from /usr/local that I haven't accounted for. I'm looking into that one now. I'm not sure how to interpret the coverage drop, though... |
|
@radarhere I've updated the pin for multibuild to include the (just merged) PR reverting the libjpeg-turbo CMAKE_INSTALL_LIBDIR change; and I've addressed some additional locations where Homebrew can leak in on x86_64. |
This has already happened many times in the past, so thanks for working on this.
I'm not sure if I understand what you mean here. We need to use a vendored version of raqm in wheels for license reasons. When using |
Thanks - that's some helpful context. I was seeing the missing fribidi detection during the build and interpreting that as something that was part of the move away from homebrew; but I see now that Pillow is loading from Homebrew by design in this instance. That also might explain the coverage drop (as supplying fribidi means any runtime loading code won't be in use). I'll take another swing at that part (and resolve the linux build issues as well). Out of interest - what's the license issue with raqm? The source code indicates It looks to be MIT licensed... did it used to be GPL (or something else problematic?) |
From #2753 in 2017:
FriBiDi is LGPL. Since then, Raqm added support for Apache-licensed SheenBiDi in 2021: HOST-Oman/libraqm#138, HOST-Oman/libraqm#139. Perhaps we could switch. |
|
@radarhere After a lot of false starts (apologies for the noise, BTW...), I think I've finally got it all sorted.
|
|
I'm not sure I understand how coverage can go down when you don't touch any code or tests... and the codecov report isn't really helping explain what is going on. |
|
Hmm, let's see... (Short version: I think it's some problem with Codecov and not this PR.) Codecov detailsLooking at the coverage report at https://github.com/python-pillow/Pillow/pull/8497/checks?check_run_id=32242849880 it says "90.36% (-3.98%) compared to 2d1d801" which is already suspicious: our coverage is around 90%, not 94%. That 2d1d801 is the last commit on That "[ci skip]" did in fact skip the CI, but I see @radarhere later triggered a manual build of that commit at https://github.com/python-pillow/Pillow/actions/runs/11573601676 for only Windows, presumably checking the PyPy failure was also happening on Anyway, this Windows-only CI resulted in a skewed 94% coverage report at https://app.codecov.io/github/python-pillow/Pillow/commit/2d1d801ec032bfb760bede88da6dcaab5ff3c75e (see the flags on the right only show Windows and not the others). All of which suggests this is nothing to worry about. Having said that, these jobs ran about 9 hours ago and @freakboy3742's comment is about 14 hours ago, so this digging doesn't reflect the situation 14 hours ago! I will say that we've been seeing drops in other PRs where it doesn't quite make sense. I've not had time to investigate, I wonder if we're not setting tokens properly for some builds after Codecov changed their v4 action to be more strict. |
|
@hugovk /me casually drops this article into the chat :-) FWIW - BeeWare ditched Codecov a couple of years ago specifically because of weird inconsistencies like this one; and although the output of a raw coverage report is marginally less pretty (i.e., no annotated code listings), you can get the same report locally that you get in CI, and you get none of the weird outages and reporting issues that seem to plague Coverage. YMMV; but if there's interest, it's a relatively straightforward switch, especially if coverage reporting is already in place. |
Co-authored-by: Andrew Murray <[email protected]>
|
The (seemingly inconsistent) Windows/pypy3.10 test failure also fascinates me... is this the same cache issue mentioned by #8513? |
.github/workflows/wheels-test.sh
Outdated
| # However, we *do* need Homebrew to provide a copy of fribidi for | ||
| # testing purposes so that we can verify the fribidi shim works as expected. | ||
| if [[ "$(uname -m)" == "x86_64" ]]; then | ||
| HOMEBREW_HOME=/usr/local/homebrew |
There was a problem hiding this comment.
| HOMEBREW_HOME=/usr/local/homebrew | |
| HOMEBREW_HOME=/usr/local |
I would have thought just '/usr/local', based on https://docs.brew.sh/Installation
The script installs Homebrew to its default, supported, best prefix (/opt/homebrew for Apple Silicon, /usr/local for macOS Intel
There was a problem hiding this comment.
Turns out both work (for brew at least) - Homebrew keeps the "original" versions in /usr/local/homebrew. However, I agree that /usr/local is the generally intended entry point.
There was a problem hiding this comment.
I take that back - it does matter (and making this change caused test failures on x86_64).
Firstly, it's possible for homebrew to have packages installed, but not linked. Using the /usr/local/Homebrew location ensures that the binary can be used regardless (note also the capitalisation fix - doesn't strictly matter on HFS+, but for accuracy).
Secondly, it's possible for other libraries to be in /usr/local/lib. Using /usr/local/Homebrew/lib for the DYLD_LIBRARY_PATH avoids this.
freakboy3742
force-pushed
the
homebrew-isolation
branch
from
4e0312c
to
51e3623
Compare
Thanks, that looks better. There still seems to be something wrong with freetype on arm64, https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:14308 But it was built with brotli, https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:7768: We need to support this for #6554. We could also consider compiling libwebp before libtiff so that WEBP-compressed tiff files can be read (AFAIK webp only looks for libtiff so that it can compile companion tools, but does not actually use it in the library - as I mentioned in #6562). But we don't need to do it now since the test was already being skipped on macOS and Linux before this PR. |
Weird. My immediate guess is that it might have something to do with the fact that a vendored version of fribidi is being used, which might not be triggering brotli support (or at least not triggering the test). I'll take a look and see what is going on.
I saw that, and ultimately gave up trying to fix it - firstly because I was trying to limit the scope of the changes, but also because I'm not sure it can be fixed. libwebp has a dependency on tiff... but then lcms2 needs tiff; and openjpeg needs libpng, tiff, and lcms2. So - I'm not sure this can be fixed (or, at least, it can't be fixed without some sort of 2-pass compile and the use of dynamic linking, which iOS can't use anyway). |
The test skip is triggered by a specific error raised by the freetype library which itself does not use fribidi.
AFAIK all of these dependencies are just for the binary utilities that we don't need so we can disable them with relevant compile flags; the libraries themselves do not have dependencies between them (except for libtiff optionally using libwebp). But I agree that it might be best to leave that for a separate PR. |
|
@nulano I've found the culprit - Homebrew's copy of freetype was leaking into the test environment. By using This can be fixed by pointing the The update includes 2 other small changes:
|
|
Thanks, looks good to me now. |
Co-authored-by: Andrew Murray <[email protected]>
Co-authored-by: Andrew Murray <[email protected]>
| @@ -448,7 +454,7 @@ def _remove_extension(self, name: str) -> None: | |||
| def get_macos_sdk_path(self) -> str | None: | |||
| try: | |||
| sdk_path = ( | |||
| subprocess.check_output(["xcrun", "--show-sdk-path"]) | |||
| subprocess.check_output(["xcrun", "--show-sdk-path", "--sdk", "macosx"]) | |||
There was a problem hiding this comment.
What was the reason for this change?
The existing macOS
cibuildwheelconfiguration relies on Homebrew to provide some dependencies. This clearly works in practice, but there are some issues associated with this choice.Firstly, it is destructive when used on a local build machine. The wheel dependency script invokes
brew remove --ignore-dependencies(which can leave the host machine in a broken, and potentially difficult to restore state); and requires the use ofsudoto make changes in the/usr/localtree. This means it is difficult to testcibuildwheelchanges locally, or to recommend usingcibuildwheelas a local build solution.Secondly, Homebrew builds could be incompatible with Pillow builds. The
MACOSX_DEPLOYMENT_TARGETfor Homebrew dependencies is fixed by the Homebrew build chain, rather than the Pillow build tools, so a change in Pillow's configuration may not be satisfied by Homebrew's build configuration. This isn't an issue at present, but it could easily become one in future.Related to this, there is also the potential that a change in a Homebrew recipe could lead to the inadvertent introduction of additional binary libraries into the delocated macOS binaries. This is especially problematic with the fribidi, raqm and freetype dependedencies which are included as "vendor" sources.
However, the real motivation for proposing this change is that it is a precursor for PEP 730-compliant iOS builds. If Homebrew is on the build path when compiling iOS binaries, compiler tooling will often find an ARM64 Homebrew binary and attempt link it into an iOS library - which then (predictably) breaks. So - Homebrew isolation is essential for iOS compilation. Since the build processes for iOS and macOS are very similar, and Homebrew isolation was necessary for iOS, adding Homebrew isolation for macOS will simplify any future iOS patch. It also provides a way to submit iOS support in a series of smaller pieces, rather than a single "monster" patch. Plus, it provides all the side benefits of isolation described above.
An overview of the changes:
Uses the
buildfolder as a location for all dependency builds. This prevents the root checkout from becoming dirtied by dependency artefacts, simplifying cleanup, and avoiding the need for extensive modifications to the.gitignorefile.Sets up an isolated build prefix in the
buildfolder (./build/deps) where dependencies can be installed, andmake installsall dependencies into that path.Forces
PATHto be a "clean" environment that only includes bare system tools, plus the Python binary being used for the build, and the isolated build prefix. macOS doesn't include most of it's development libraries in /usr or /usr/local, instead using a path provided by the macOS SDK (which is configured as part of the compiler toolchain).Adds a build of
pkg-configto the dependencies. This is the one build tool that autotools and cmake often require that Xcode doesn't provide. Providing a custom build ofpkg-configalso ensures that no dependencies other than the ones provided by cibuildwheel are included. The build recipe that is used is derived from the Homebrew recipe.Adds a build of
libXdmcp, to avoid a dependency on the Homebrew-provided version.Adds builds offribidiandraqm, rather than using the Homebrew versions as "vendored" versions. Thesetup.pyconfiguration passed in bycibuildwheelhas been modified from the default on other platforms to not use thevendorversion.Updates the pinned multibuild version to the current
develbranch hash. This is to incorporate Python3.13 support, plus a number of fixes required to support installs into locations other than/usr/local, and corrections to LIBDIR handling for platforms that use thelib64suffix.Adds a
dependencies-prefixconfiguration option tosetup.py. This allows the user to provide a specific location to look for dependencies, rather than the series of Fink/MacPorts/Homebrew fallbacks that currently exist.