Workflow to build python CLI artifacts (sdist + wheels) #3036

HonahX · 2025-11-13T01:06:25Z

This PR include a github workflow, "Python Client Build Artifacts", to build wheels for multiple os/platform for nightly build and release automation in the future. It will produce a zip containing
contain sdist and wheels built for macos and linux (TODO: support windows)

> tar -tzf python-client-artifacts-20141056849.zip
apache_polaris-1.2.0-cp310-cp310-macosx_14_0_arm64.whl
apache_polaris-1.2.0-cp310-cp310-manylinux_2_39_x86_64.whl
apache_polaris-1.2.0-cp310-cp310-macosx_15_0_arm64.whl
apache_polaris-1.2.0-cp311-cp311-macosx_14_0_arm64.whl
apache_polaris-1.2.0-cp311-cp311-macosx_15_0_arm64.whl
apache_polaris-1.2.0-cp311-cp311-manylinux_2_39_x86_64.whl
apache_polaris-1.2.0-cp312-cp312-macosx_14_0_arm64.whl
apache_polaris-1.2.0-cp312-cp312-macosx_15_0_arm64.whl
apache_polaris-1.2.0-cp312-cp312-manylinux_2_39_x86_64.whl
apache_polaris-1.2.0-cp313-cp313-macosx_14_0_arm64.whl
apache_polaris-1.2.0-cp313-cp313-macosx_15_0_arm64.whl
apache_polaris-1.2.0-cp313-cp313-manylinux_2_39_x86_64.whl
apache_polaris-1.2.0.tar.gz

The expected usage for this workflow is:

Triggered for every python client changes to verify that changes will not break wheel building
Called by other workflow (e.g. nightly build) like

....
  build-nightly-artifacts:
    needs:
      - generate-version
    uses: ./.github/workflows/python-client-build-artifacts.yml
    with:
      version: ${{ needs.generate-version.outputs.VERSION }}

....
-- download artifacts

-- upload to test.pypi.org

....

Called by maintainer directly for test purpose.

The build of a release candidate that include proper sig, checksum, is out-of-scope of this PR and should be included in follow-up PR for python client release automation

cc: @binarycat0 @MonkeyCanCode @flyrain @snazy

Checklist

🛡️ Don't disclose security issues! (contact [email protected])
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

client/python/generate_clients.py

.github/workflows/python-client-release-pypi-artifact.yml

.github/workflows/python-client-build-artifacts.yml

.github/workflows/python-client-release-svn-artifact.yml

.github/workflows/python-release-build.yml

MonkeyCanCode · 2025-11-19T05:05:15Z

@HonahX very nice work. We are almost there to have package in PyPI.

snazy

Is there a particular reason to not extent the release-automation workflows?

We have to follow the ASF release policy. It seems there are some steps missing in this PR:

checksums
gpg signatures
RC staging
staging the source tarball and artifacts to vote on
eventually publishing exactly the artifacts that have been voted on

HonahX · 2025-11-20T03:30:28Z

Hi @snazy Thanks for highlighting these important elements of the final release candidate for python CLI. Given the number of missing pieces in the CLI release process, my goal is to separate the pipeline into 2 milestone:

Have a formalized way to build release artifacts (wheels) that will later be released to PyPI for users to install.
Have a formalized way to build and upload release candidate that include proper signature and checksum of release artifacts (release automation pipeline)

This PR aims to focus on the first milestone in order to have additional CI to ensure that all changes are compatible with the release process and unblock nightly build. The signature/checksum/release candidate are goals in the second milestone. This way we could first determine what to release and start testing the artifacts and then finishing the rest part. WDYT?

snazy · 2025-11-26T04:51:04Z

@HonahX is this PR a draft PR or not? The summary says yes, the PR status says no.

HonahX · 2025-11-26T05:46:27Z

Hi @snazy It's ready for review. I checked the current PR description and it does not seem to imply this as a draft. Could you please point me to the part of the summary that may lead to the wrong implication here? Just want to eliminate all potential confusion.

snazy

The PR adds completely new release workflows for separate Python releases.

We already have release workflows in place, so I do not see a point having a separate set of workflows.

Technically it's missing a lot of the requirements for release artifacts for ASF projects and the ASF release process.

A huge amount of time and work already went into the existing release automation, because it is not an easy task to get it right.

HonahX · 2025-12-02T03:24:56Z

Hi @snazy, I think there might be some confusion in the original PR summary and the workflow name. I should not put the "release" in the workflow title as the intention was not to make this a separate release procedure than the existing release automation. Ultimately, the python client release process should be part of the release automation.

I've updated the PR to simplify the workflow file and updated the summary to clarify the intention. We will focus on build the right wheels and sdist and make nightly build working first. Sorry for any confusion above. Please take a look again and let me know WDYT!

snazy · 2025-12-02T18:09:37Z

I can take another look, but due to time constraints likely not before next week.

flyrain · 2025-12-04T02:24:24Z

.github/workflows/python-client-build-artifacts.yml

+        uses: pypa/[email protected]
+        with:
+          output-dir: wheelhouse
+          package-dir: "./client/python"
+          config-file: "{package}/pyproject.toml"
+        env:
+          # Ignore 32 bit architectures
+          CIBW_ARCHS: "auto64"
+          CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10,<3.14"
+          CIBW_REPAIR_WHEEL_COMMAND_MACOS: "" # Disable wheel repair since there will be no binary in the wheel
+          CIBW_REPAIR_WHEEL_COMMAND_LINUX: ""
+          CIBW_ENVIRONMENT: POLARIS_CLI_SKIP_CLIENT_GENERATION="true" # java is not available in linux containers during wheel build, skip client generation


Curious: does this command archive the same result?

run: | make client-build FORMAT=wheel

It should be mostly the same for MacOs/windows, but for linux it will help build both manylinux and musllinux wheel to have broad compatibility.

flyrain

LGTM. Thanks @HonahX !

snazy

I think we do not need the overhead of CPython.
Without that, I suspect the Python package build can be moved into the Makefile and the calling workflows would just need to setup the necessary Python versions.

Then the existing nightly-build workflow can be extended with one job to setup Python, call that Makefile target and publish to test.pypi.org.

snazy · 2025-12-08T10:39:07Z

.github/workflows/python-client-build-artifacts.yml

+          make client-build FORMAT=sdist
+
+      - name: Build wheels
+        uses: pypa/[email protected]


Actions in ASF workflows must reference the Git SHA of an ASF-approved version of the action, see this policy.

The action docs mention that the wheels can contain libraries from the build container.
Which ones are included and what are the licenses for those libraries?

Good catch. We definitely should use the approved version here: https://github.com/apache/infrastructure-actions/blob/7a58482c11f14b79ea238d4a57093077cc18673c/approved_patterns.yml#L174C3-L174C61

The action docs mention that the wheels can contain libraries from the build container.

The dependencies may be brought by the wheel repair operation. In our case, we explicitly disabled the wheel repair:

CIBW_REPAIR_WHEEL_COMMAND_MACOS: "" CIBW_REPAIR_WHEEL_COMMAND_LINUX: ""

so no additional dependencies will be brought in.

But based on other comments and discussion, cibuildwheel may not be the best way here, let me try another approach first

Switched to use make client-build FORMAT=wheel directly instead of cibuildwheel

snazy · 2025-12-08T10:44:12Z

.github/workflows/python-client-build-artifacts.yml

+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version to set (optional, e.g. 1.1.0 or 1.1.0rc1 or dev-20251202-abc1234)'
+        type: string
+        required: false
+  workflow_call:
+    inputs:
+      version:
+        description: 'Version to set (optional)'
+        type: string
+        required: false
+  pull_request:
+    paths:
+      - '.github/workflows/python-client-build-artifacts.yml'
+      - 'client/python/**'


I thought this WF is just for nightly builds?
I'm confused about the need for a "version" input and that the WF is triggered by PRs.

The version here is optional and used to set the package version here

- name: Set version (if provided) if: inputs.version != '' env: VERSION: ${{ inputs.version }} run: make client-set-version VERSION="${{ env.VERSION }}"

In the nightly build, we will need to determine a convention of the package version for nightly build, such as including commit hash/date to the package version, so we could have a better reference in test.pypi.org. But IMHO the details of the version we want to set to the package could be determined in the actual nightly build publication step. This workflow should focus on building all the wheels required, by default leave the package version unchanged and make it an option for any upstream caller to set the version as needed. WDYT?

snazy · 2025-12-08T10:46:22Z

.github/workflows/python-client-build-artifacts.yml

+    strategy:
+      matrix:
+        os: [ ubuntu-24.04, macos-14, macos-15] # current build script is not compatible with windows


Is there anything that's performance critical in the Python package?
I'd prefer to not build native code.

Yeah, ideally we only need a universal wheel since everything is pure python. However, since we need to leverage a custom build script to do open api client generation, poetry thinks it needs to generate platform-specific wheels. This is a known limitation and seems we have no native way to override this behavior now.

Should the situation changed in the future, we should definitely switch to universal wheel if possible.

snazy · 2025-12-08T10:47:41Z

.github/workflows/python-client-build-artifacts.yml

+          if [ -n "${{ inputs.version }}" ]; then
+            ARTIFACT_NAME="python-client-artifacts-${{ inputs.version }}"


What's the reason for this special case?

It could make the output zip file name indicate that a custom version has been provided hence the built sdist and wheels will have its package version changed. Just for convenience.

client/python/generate_clients.py

.github/workflows/python-client-build-artifacts.yml

github-project-automation bot added this to Basic Kanban Board Nov 13, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Nov 13, 2025

HonahX linked an issue Nov 14, 2025 that may be closed by this pull request

Python CLI Publication: Github workflow to build artifacts #3047

Open

HonahX changed the title ~~[Draft] Workflows to build python CLI release and pypi artifacts~~ Workflows to build python CLI release and pypi artifacts Nov 19, 2025

HonahX commented Nov 19, 2025

View reviewed changes

client/python/generate_clients.py Outdated Show resolved Hide resolved

HonahX marked this pull request as ready for review November 19, 2025 04:41

MonkeyCanCode self-requested a review November 19, 2025 04:48

MonkeyCanCode reviewed Nov 19, 2025

View reviewed changes

snazy reviewed Nov 19, 2025

View reviewed changes

HonahX force-pushed the honahx-python-release-2 branch from 9247338 to 36a45e4 Compare November 25, 2025 21:41

snazy requested changes Nov 26, 2025

View reviewed changes

HonahX changed the title ~~Workflows to build python CLI release and pypi artifacts~~ Workflows to build python CLI artifacts (sdist + wheels) Dec 2, 2025

HonahX changed the title ~~Workflows to build python CLI artifacts (sdist + wheels)~~ Workflow to build python CLI artifacts (sdist + wheels) Dec 2, 2025

HonahX requested review from MonkeyCanCode, binarycat0 and snazy December 2, 2025 17:34

flyrain reviewed Dec 4, 2025

View reviewed changes

flyrain previously approved these changes Dec 4, 2025

View reviewed changes

MonkeyCanCode previously approved these changes Dec 4, 2025

View reviewed changes

snazy reviewed Dec 8, 2025

View reviewed changes

HonahX dismissed stale reviews from MonkeyCanCode and flyrain via 7facfa4 December 11, 2025 17:01

HonahX added 2 commits December 11, 2025 12:09

Add artifact build pipeline

d30d9fa

fix license

e67bfcb

HonahX added 5 commits December 11, 2025 12:09

Fix format

3c53fbf

Fix nit and format issue

2b07c3e

Simplify build file

f2331e0

Remove the need of cibuildwheel

6c5b7f3

Remove unnecessary step

2578b89

HonahX force-pushed the honahx-python-release-2 branch from abe51b0 to 2578b89 Compare December 11, 2025 18:09

HonahX requested a review from snazy December 11, 2025 18:10

		if [ -n "${{ inputs.version }}" ]; then
		ARTIFACT_NAME="python-client-artifacts-${{ inputs.version }}"

Workflow to build python CLI artifacts (sdist + wheels) #3036

Are you sure you want to change the base?

Workflow to build python CLI artifacts (sdist + wheels) #3036

Conversation

HonahX commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MonkeyCanCode commented Nov 19, 2025

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

HonahX commented Nov 20, 2025

Uh oh!

snazy commented Nov 26, 2025

Uh oh!

HonahX commented Nov 26, 2025

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

HonahX commented Dec 2, 2025

Uh oh!

snazy commented Dec 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HonahX Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HonahX commented Nov 13, 2025 •

edited

Loading

HonahX Dec 11, 2025 •

edited

Loading