cibuildwheel: Add macOS arm64 support

[xavier2k6]

Closes #7308.
Closes #7473.
Closes #7536.
Closes #7650.
Closes #7683.
Closes #7729.

[xavier2k6]

Note/Review: pypa/cibuildwheel#1926 (comment)

[xavier2k6]

@qstokkink Can you give the created wheels a test?

[xavier2k6]

Upcoming changes:

• Support for musllinux_1_1 images will be dropped on 2024-11-01
• feat: use SPEC 0 schedule for cibuildwheel

[qstokkink]

@qstokkink Can you give the created wheels a test?

Works on my machine. 👍

On behalf of the Tribler team, I would like to request you to keep {"os": "macos-12", "CIBW_BUILD": "cp39-*", "CIBW_ARCHS": "x86_64"} in the workflow dispatch matrix though.

[xavier2k6]

@qstokkink

1. I've added macOS-12/x86_x64 architecture with Python 3.9 to pull_request matrix for testing purposes (this can probably be removed again, once you've tested)
2. Have included macOS-12/x86_x64 architecture with Python * to workflow dispatch unless of course You/Tribler team only want Python 3.9/x86_x64 on macOS-12??

[qstokkink]

Thanks 👍 and, "correct" to 2: I copied the wrong string.

[xavier2k6]

@qstokkink Just so the Tribler team are aware macOS-12 images are considered Deprecated & will be Removed by end of year!
See: actions/runner-images#9255 or screenshot below.

---

[qstokkink]

🤔 that's sad. It would be nice to get a PyPI release before that gets killed though.

[qstokkink]

@xavier2k6 considering the other closed issues, it looks like you can add #7729 to your "Closes" list.

[xavier2k6]

manylinux2014 image may be dropped in a upcoming PR too by "cibuildwheel" repo., so may need to change to manylinux_2_28 but this may also drop older python versions but that is already on the cards by "cibuildwheel" repo.

[xavier2k6]

@arvidn Regarding to OpenSSL 1.1.1 End Of Life

---

1. Should wheels for Python <3.10 be dropped?

• Python 3.11: - Windows builds and macOS installers from python.org now use OpenSSL 3.0.
• feat: use SPEC 0 schedule for cibuildwheel

---

2. What minimum OpenSSL 3.x.x version should be declared for below?

libtorrent/tools/cibuildwheel/manylinux/build-openssl.sh

Line 19 in 2ab8fed

OPENSSL_MIN_VERSION=1.1.1

libtorrent/tools/cibuildwheel/manylinux/openssl-version.sh

Lines 3 to 4 in 2ab8fed

	export OPENSSL_ROOT=openssl-1.1.1l
	export OPENSSL_HASH=0b7a3e5e59c34827fe0c3a74b7ec8baef302b98fa80088d7f9153aa16fa76bd1

---

3. FORTIFY_SOURCE may be bumped to 3 if images are newer/all conditions/requirements are met.

libtorrent/tools/cibuildwheel/manylinux/build_utils.sh

Line 8 in 2ab8fed

MANYLINUX_CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2"

[xavier2k6]

@arvidn I'll leave the OpenSSL 32bit installation to you, do you still want to support 32bit??

We also now have a definitive timeline for the deprecation of the macOS-12 runner
The macOS 12 Actions runner image will begin deprecation on 10/7/24 and will be fully unsupported by 12/3/24

[qstokkink]

@xavier2k6 Do you have time to experiment a bit with the win32 build? If we're lucky, it could be as simple as inserting the following at line 106 of cibuildwheel.yml:

- name: Install OpenSSL
  if: ${{ endsWith(matrix.CIBW_BUILD, 'win32') }}
  run: choco install openssl

If all checks are green/passing, it would be easier to merge this PR, I guess.

[xavier2k6]

@qstokkink i'll be able to give that a try in about 3hrs (got a spare few mins now to do it), action needs to be updated too & python 3.13 is gone GA.

[xavier2k6]

vcpkg is on the windows runners, this could be used to install openssl perhaps.....

[qstokkink]

@xavier2k6 Thanks for putting some time into this. Using vcpkg should also work, but the current setup was made for chocolatey:

libtorrent/Jamfile

Lines 442 to 446 in 790b662

	# the de-facto windows installer is https://slproweb.com/products/Win32OpenSSL.html, which installs to c:\Program Files\OpenSSL-Win{32,64}.
	# chocolatey appears to use this installer.
	local address_model = [ feature.get-values <address-model> : $(properties) ] ;
	OPENSSL_LIB += "C:/Program Files/OpenSSL-Win$(address_model)/lib" ;
	OPENSSL_LIB += "C:/Program Files (x86)/OpenSSL-Win$(address_model)/lib" ;

libtorrent/Jamfile

Lines 473 to 474 in 790b662

	OPENSSL_INCLUDE += "C:/Program Files/OpenSSL-Win$(address_model)/include" ;
	OPENSSL_INCLUDE += "C:/Program Files (x86)/OpenSSL-Win$(address_model)/include" ;

---

Using your log output, switching the four paths to this should work:

if address_model = 32
{
    OPENSSL_LIB += "C:/vcpkg/packages/openssl_x86-windows-static-md/lib" ;
} else {
    OPENSSL_LIB += "C:/vcpkg/packages/openssl_x64-windows-static-md/lib" ;
}

if address_model = 32
{
    OPENSSL_INCLUDE += "C:/vcpkg/packages/openssl_x86-windows-static-md/include" ;
}  else {
    OPENSSL_INCLUDE += "C:/vcpkg/packages/openssl_x64-windows-static-md/include" ;
}

My Jam language is a bit rusty. So, this might need to be address_model = "32".

[xavier2k6]

@qstokkink Will make some changes later, will also need to make sure we use the right package as there's 3x release options I believe:
openssl:x86-windows-static
openssl:x86-windows-static-md
openssl:x86-windows

I'll have to look at vcpkg port/json files etc.

vcpkg gets updated each time there's a newer windows runner image released so that will bring newer versions of OpenSSL.

[qstokkink]

Just for motivation/verification: the macos-14 wheel (from this PR) is now part of the Tribler unit tests and everything is working correctly.

[xavier2k6]

I'm not too sure if the Jamfile is the right approach as we only need vcpkg for the OpenSSL install in CI not as an overall change.

[xavier2k6]

@arvidn Do you still plan/want to support 32bit?

[xavier2k6]

@qstokkink @arvidn the 32bit OpenSSL issue is the only thing stopping this PR from being finished.....any help would be highly appreciated.

I could do a dry-run then (if really necessary) for all python versions/OS before marking for review.

[qstokkink]

@xavier2k6 I can help with that. Are you O.K. with me opening another PR that includes your commit?

[xavier2k6]

@qstokkink Feel free! I would suggest however to remove all other OS's from Pull Request section & just have Windows x86 & Windows x64 for testing purposes of OpenSSL....this will reduce build time/waiting on runners etc...we know the other OS's work & they can be added in via a final push.

[qstokkink]

Ok, maybe a separate PR is overkill. This is all you need:

Suggested change

	- name: Install OpenSSL (win32)
	if: ${{ endsWith(matrix.CIBW_BUILD, 'win32') }}
	run: |
	Remove-Item -Path "C:\Program Files\OpenSSL" -Force -Recurse
	vcpkg install openssl:x86-windows
	New-Item -Path "C:\Program Files\OpenSSL" -ItemType SymbolicLink -Value "C:\vcpkg\packages\openssl_x86-windows\"

There's a mini build here: https://github.com/qstokkink/libtorrent/actions/runs/11439039359

[xavier2k6]

@qstokkink Thank you!, added your change now.

[xavier2k6]

@qstokkink Do you think we should use the same vcpkg method for x64? (consistency)

[qstokkink]

In my opinion: absolutely not. This is much slower (it adds 6~7 minutes to the job). That is way too much added time just for the sake of consistency.

[arvidn]

thanks everyone!

[zakkarry]

@arvidn would it be possible before support for 1.x musl and any other support is dropped to get a pypi build for 3.12 of 1.2.19 published as well?