gh-91349: Replace zlib with zlib-ng in Windows build #131438

zooba · 2025-03-19T01:11:08Z

Issue: Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

zooba · 2025-03-19T01:12:37Z

Misc/externals.spdx.json

      "externalRefs": [
        {
          "referenceCategory": "SECURITY",
-          "referenceLocator": "cpe:2.3:a:zlib:zlib:1.3.1:*:*:*:*:*:*:*",
+          "referenceLocator": "cpe:2.3:a:zlib-ng:zlib-ng:2.2.4:*:*:*:*:*:*:*",


I'll admit I just guessed this, in order to unblock my testing (couldn't build at all with an invalid SBOM). If it's not right, let me know

This is likely the CPE that would get used, although we can't be for certain until a CVE exists with it... the only one I could find with some searching is cpe:...:zlib-ng:minizip-ng which is for a different component but at least the organization is a match.

zooba · 2025-03-19T01:13:14Z

Modules/zlibmodule.c

@@ -2101,6 +2101,12 @@ zlib_exec(PyObject *mod)
                     PyUnicode_FromString(zlibVersion())) < 0) {
        return -1;
    }
+#ifdef ZLIBNG_VERSION
+    if (PyModule_Add(mod, "ZLIBNG_VERSION",


Figured we'd want some way to detect this other than looking at the text in ZLIB_VERSION

makes sense. annoying to have optional attributes that one would use hasattr or getattr on instead of blindly accessing, but realistically nobody should care as this is more internal informational so this is fine.

Yeah, the main place it'll be used is in pythoninfo, which already handles absent attributes.

We have more offensive optional attributes in our extension modules 😆

zooba · 2025-03-19T01:24:45Z

I've reviewed the three header files added to PC and they seem generic. It's a shame they aren't part of the sources, but figured it's easier/safer to just copy them into our source tree so that the cpython-source-deps repo is a straight import.

bedevere-bot · 2025-03-19T01:35:18Z

🤖 New build scheduled with the buildbot fleet by @zooba for commit 08eecb1 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F131438%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

PCbuild/pythoncore.vcxproj

zooba · 2025-03-19T13:45:34Z

The two failing buildbots appear unrelated, and everything else passed. All I changed in the last commit was docs, so I'm not going to rerun the buildbots unless someone thinks it's necessary or worthwhile.

sethmlarson

SBOM changes LGTM!

sethmlarson · 2025-03-19T13:38:26Z

Misc/externals.spdx.json

      "externalRefs": [
        {
          "referenceCategory": "SECURITY",
-          "referenceLocator": "cpe:2.3:a:zlib:zlib:1.3.1:*:*:*:*:*:*:*",
+          "referenceLocator": "cpe:2.3:a:zlib-ng:zlib-ng:2.2.4:*:*:*:*:*:*:*",


This is likely the CPE that would get used, although we can't be for certain until a CVE exists with it... the only one I could find with some searching is cpe:...:zlib-ng:minizip-ng which is for a different component but at least the organization is a match.

mdboom · 2025-03-19T14:22:55Z

I'm kicking off a pyperformance benchmarking run with this -- we have a non-zero usage of zlib in that suite.

mdboom · 2025-03-19T18:02:38Z

The results for this on pyperformance are basically within the noise / no change. Not all that surprising, given that there are no benchmarks specifically aimed at zlib. https://github.com/faster-cpython/benchmarking-public/tree/main/results/bm-20250319-3.14.0a6+-548daa7

gpshead · 2025-03-19T18:21:32Z

Modules/zlibmodule.c

@@ -2101,6 +2101,12 @@ zlib_exec(PyObject *mod)
                     PyUnicode_FromString(zlibVersion())) < 0) {
        return -1;
    }
+#ifdef ZLIBNG_VERSION
+    if (PyModule_Add(mod, "ZLIBNG_VERSION",


makes sense. annoying to have optional attributes that one would use hasattr or getattr on instead of blindly accessing, but realistically nobody should care as this is more internal informational so this is fine.

chris-eibl · 2025-03-20T16:57:13Z

This has broken the tailcall builds on Windows: https://github.com/python/cpython/actions/runs/13967899073/job/39102464260

D:\a\cpython\cpython\externals\\zlib-ng-2.2.4\\functable.c(79,26): error : use of undeclared identifier 'slide_hash_sse2'; did you mean 'slide_hash_c'? [D:\a\cpython\cpython\PCbuild\zlib-ng.vcxproj]
D:\a\cpython\cpython\externals\\zlib-ng-2.2.4\\functable.c(123,26): error : use of undeclared identifier 'slide_hash_avx2'; did you mean 'slide_hash_c'? [D:\a\cpython\cpython\PCbuild\zlib-ng.vcxproj]

zooba · 2025-03-20T17:01:30Z

Probably needs some fixed preprocessor checks (hopefully only in what's in the PC directory) to better handle the intrinsics being present/absent.

Though I notice that build warns NOTE: Visual Studio not detected. LLVM does not provide a C/C++ standard library and may be unable to locate MSVC headers., which might also be relevant.

chris-eibl · 2025-03-20T17:24:07Z

I think, this is because here
https://github.com/zlib-ng/zlib-ng/blob/860e4cff7917d93f54f5d7f0bc1d0e8b1a3cb988/fallback_builtins.h#L4-L25
in line 4 __clang__ gets excluded and hence #define HAVE_BUILTIN_CTZ is not set (but in the MSVC case it is).

Then, in
https://github.com/zlib-ng/zlib-ng/blob/860e4cff7917d93f54f5d7f0bc1d0e8b1a3cb988/arch/x86/x86_functions.h#L13-L20
we dont get the prototype for slide_hash_sse2 in line 17, hence the error.

Sorry, I cannot get the permalinks to work here the usual way I am doing it :(

chris-eibl · 2025-03-20T17:30:51Z

Though I notice that build warns NOTE: Visual Studio not detected. LLVM does not provide a C/C++ standard library and may be unable to locate MSVC headers., which might also be relevant.

FWIW, when building with the bundled clang-cl of VS I do not have these warnings, but get the same error.

zooba · 2025-03-20T17:37:20Z

Most likely zlib-ng would generate different versions of the header files I checked into the PC directory for Clang. Figuring out what differences belong in there and detecting it dynamically rather than using CMake is the way forward.

This should get a new issue (because clang-cl isn't a supported configuration, so we don't have to revert this change until it's working). Tag me on it and we can continue there.

chris-eibl · 2025-03-20T17:44:41Z

PCbuild/zlib-ng.vcxproj

+      <PrecompiledHeader>NotUsing</PrecompiledHeader>
+      <AdditionalIncludeDirectories>$(zlibNgDir);$(PySourceDir)PC;$(GeneratedZlibNgDir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <PreprocessorDefinitions>%(PreprocessorDefinitions);ZLIB_COMPAT;WITH_GZFILEOP;NO_FSEEKO;HAVE_BUILTIN_ASSUME_ALIGNED;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;</PreprocessorDefinitions>
+      <PreprocessorDefinitions Condition="$(Platform) == 'Win32' or $(Platform) == 'x64'">%(PreprocessorDefinitions);X86_FEATURES;X86_HAVE_XSAVE_INTRIN;X86_SSE2;X86_SSSE3;X86_SSE42;X86_PCLMULQDQ_CRC;X86_AVX2;X86_AVX512;X86_AVX512VNNI;X86_VPCLMULQDQ_CRC</PreprocessorDefinitions>


Does this mean that for all executables X86_AVX512 code will be emitted and used?
See e.g. adler32_avx512.c, where the whole file is guarded with X86_AVX512 and there is code like __m512i in it.

I believe it's detected and called dynamically based on availability, but it's all compiled at compile time.

Yeah, I hope so, too.

chris-eibl · 2025-03-20T20:34:53Z

PCbuild/zlib-ng.vcxproj

+      <PreprocessorDefinitions>%(PreprocessorDefinitions);ZLIB_COMPAT;WITH_GZFILEOP;NO_FSEEKO;HAVE_BUILTIN_ASSUME_ALIGNED;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;</PreprocessorDefinitions>
+      <PreprocessorDefinitions Condition="$(Platform) == 'Win32' or $(Platform) == 'x64'">%(PreprocessorDefinitions);X86_FEATURES;X86_HAVE_XSAVE_INTRIN;X86_SSE2;X86_SSSE3;X86_SSE42;X86_PCLMULQDQ_CRC;X86_AVX2;X86_AVX512;X86_AVX512VNNI;X86_VPCLMULQDQ_CRC</PreprocessorDefinitions>
+      <PreprocessorDefinitions Condition="$(Configuration) == 'Debug'">%(PreprocessorDefinitions);ZLIB_DEBUG</PreprocessorDefinitions>
+      <EnableEnhancedInstructionSet Condition="$(Platform) == 'Win32' or $(Platform) == 'x64'">AdvancedVectorExtensions2</EnableEnhancedInstructionSet>


Unconditionally letting the compiler generate code up to AVX2 for all *.c files here might result in the binary not running on all CPUs?

…131438)

zooba added 4 commits March 19, 2025 00:56

pythongh-91349: Replace zlib with zlib-ng in Windows build

de56164

Updated readme.txt

1347c50

Fix includes for PGO build

7fd8f26

Use correct include dirs

f919749

zooba requested a review from gpshead March 19, 2025 01:11

zooba requested review from sethmlarson and a team as code owners March 19, 2025 01:11

bedevere-app bot added the awaiting core review label Mar 19, 2025

bedevere-app bot mentioned this pull request Mar 19, 2025

Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

Open

Add NEWS

fb6500d

zooba commented Mar 19, 2025

View reviewed changes

zooba added 2 commits March 19, 2025 01:25

Fix debug reference and some warnings

89d44a1

Exclude Intel optimizations better on ARM64

08eecb1

zooba added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 19, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 19, 2025

rhpvorderman reviewed Mar 19, 2025

View reviewed changes

PCbuild/pythoncore.vcxproj Show resolved Hide resolved

Update docs and whatsnew

548daa7

sethmlarson approved these changes Mar 19, 2025

View reviewed changes

Remove unresolved reference from docs

f108044

gpshead approved these changes Mar 19, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Mar 19, 2025

Include ZLIBNG_VERSION in pythoninfo

085fa10

zooba enabled auto-merge (squash) March 19, 2025 18:38

zooba merged commit 63a638c into python:main Mar 19, 2025
42 checks passed

bedevere-app bot removed the awaiting merge label Mar 19, 2025

zooba deleted the gh-91349 branch March 19, 2025 19:25

chris-eibl reviewed Mar 20, 2025

View reviewed changes

chris-eibl mentioned this pull request Mar 20, 2025

clang-cl on Windows fails to compile after zlib-ng #131521

Closed

chris-eibl reviewed Mar 20, 2025

View reviewed changes

hugovk mentioned this pull request Mar 21, 2025

Consider distributing zlib-ng astral-sh/python-build-standalone#464

Open

chris-eibl mentioned this pull request Mar 22, 2025

GH-131521: fix clangcl build on Windows for zlib-ng #131526

Merged

zanieb mentioned this pull request Apr 9, 2025

Bump to 3.14.0a7 astral-sh/python-build-standalone#580

Open

seehwan pushed a commit to seehwan/cpython that referenced this pull request Apr 16, 2025

pythongh-91349: Replace zlib with zlib-ng in Windows build (pythonGH-…

252ef73

…131438)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-91349: Replace zlib with zlib-ng in Windows build #131438

gh-91349: Replace zlib with zlib-ng in Windows build #131438

zooba commented Mar 19, 2025 •

edited by bedevere-app bot

Loading

zooba Mar 19, 2025

sethmlarson Mar 19, 2025

zooba Mar 19, 2025

gpshead Mar 19, 2025

zooba Mar 19, 2025

zooba commented Mar 19, 2025

bedevere-bot commented Mar 19, 2025

zooba commented Mar 19, 2025

sethmlarson left a comment

sethmlarson Mar 19, 2025

mdboom commented Mar 19, 2025

mdboom commented Mar 19, 2025

gpshead Mar 19, 2025

chris-eibl commented Mar 20, 2025

zooba commented Mar 20, 2025

chris-eibl commented Mar 20, 2025

chris-eibl commented Mar 20, 2025

zooba commented Mar 20, 2025

chris-eibl Mar 20, 2025

zooba Mar 20, 2025

chris-eibl Mar 20, 2025

chris-eibl Mar 20, 2025

gh-91349: Replace zlib with zlib-ng in Windows build #131438

gh-91349: Replace zlib with zlib-ng in Windows build #131438

Conversation

zooba commented Mar 19, 2025 • edited by bedevere-app bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zooba commented Mar 19, 2025

bedevere-bot commented Mar 19, 2025

zooba commented Mar 19, 2025

sethmlarson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdboom commented Mar 19, 2025

mdboom commented Mar 19, 2025

Choose a reason for hiding this comment

chris-eibl commented Mar 20, 2025

zooba commented Mar 20, 2025

chris-eibl commented Mar 20, 2025

chris-eibl commented Mar 20, 2025

zooba commented Mar 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zooba commented Mar 19, 2025 •

edited by bedevere-app bot

Loading