Skip to content

Add Automation of Python SDK Release Process. #168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 58 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
88398fb
add wheel building action
MOmarMiraj Mar 4, 2025
8eac2b7
fix manylinux image and dont build wheels for PyPi
MOmarMiraj Mar 4, 2025
c2627e6
go to 2_34 manylinux image as theres no 2_32 image
MOmarMiraj Mar 4, 2025
ea82680
fix linux image name
MOmarMiraj Mar 4, 2025
9742677
add full url for linux image
MOmarMiraj Mar 4, 2025
16a9cb4
update setup.py to build in correct folder
MOmarMiraj Mar 4, 2025
cd8b06f
update test cmd path
MOmarMiraj Mar 4, 2025
6f659f6
checkout only the example folder
MOmarMiraj Mar 4, 2025
deaeb15
checkout example folder correctly
MOmarMiraj Mar 4, 2025
efe387e
edit path
MOmarMiraj Mar 4, 2025
670175f
add support for shared libs packaging
MOmarMiraj Mar 5, 2025
3ad27e9
add cryptography as a wheel
MOmarMiraj Mar 5, 2025
5e84ed0
make it manual
MOmarMiraj Mar 5, 2025
35d2f0d
don't stop if fail
MOmarMiraj Mar 5, 2025
67edcbb
debug working dir
MOmarMiraj Mar 5, 2025
145758d
check file permissions
MOmarMiraj Mar 5, 2025
f316654
change to python instead of python3
MOmarMiraj Mar 5, 2025
881497d
remove other jobs and see files path
MOmarMiraj Mar 5, 2025
f7a1083
try hardcoded path
MOmarMiraj Mar 5, 2025
1cc21ce
debug
MOmarMiraj Mar 5, 2025
f94d10b
see python version and go to 3.12
MOmarMiraj Mar 5, 2025
c777afd
try
MOmarMiraj Mar 5, 2025
521efd0
check here
MOmarMiraj Mar 5, 2025
bacb8c7
try bilding wheels now
MOmarMiraj Mar 5, 2025
efc4ed4
fix path of test and removei nstallation of python
MOmarMiraj Mar 5, 2025
8f19e0d
add python3
MOmarMiraj Mar 5, 2025
1a3e9bb
revert path
MOmarMiraj Mar 5, 2025
e3ffc3e
add script
MOmarMiraj Mar 5, 2025
ea0c161
add permission
MOmarMiraj Mar 5, 2025
fce070e
fix name
MOmarMiraj Mar 5, 2025
8bba2eb
delete script
MOmarMiraj Mar 5, 2025
028d32b
fix test command
MOmarMiraj Mar 5, 2025
2e6941f
pass sa token to linux
MOmarMiraj Mar 5, 2025
4e944bd
add morgan change
MOmarMiraj Mar 5, 2025
cd9b562
fix rules of when this job wil run
MOmarMiraj Mar 5, 2025
98f4832
update sa token
MOmarMiraj Mar 5, 2025
8bf9ccb
test on test_client instead of example.py
MOmarMiraj Mar 5, 2025
e88f750
skip musllinux wheels
MOmarMiraj Mar 5, 2025
976c276
fix rules for when the job is ran
MOmarMiraj Mar 5, 2025
2a09d16
check if its manual and its a release branch
MOmarMiraj Mar 5, 2025
1fc3895
add input
MOmarMiraj Mar 5, 2025
c3f0211
remove the push
MOmarMiraj Mar 5, 2025
cc5cf76
fix up yml file
MOmarMiraj Mar 5, 2025
5690c32
revert example file
MOmarMiraj Mar 5, 2025
2345ad1
clean up setup.py
MOmarMiraj Mar 5, 2025
a16e535
add build sdist and fix EOL
MOmarMiraj Mar 5, 2025
94fe548
add test job for source distro
MOmarMiraj Mar 6, 2025
1a487ca
test sdist job
MOmarMiraj Mar 6, 2025
f7b1d8c
install python dep for source distro
MOmarMiraj Mar 6, 2025
3593dfb
clean up .yml file
MOmarMiraj Mar 6, 2025
212f8a6
add publishing of pypi to workflow
MOmarMiraj Mar 6, 2025
fa1c3ec
fix wheels to correctly build and publish wheels to pypi and add pypr…
MOmarMiraj Mar 8, 2025
314cdc7
add new line at EOF
MOmarMiraj Mar 8, 2025
a004bf2
Add prep-release and release and update wheels according to fork of P…
MOmarMiraj Apr 3, 2025
43548d1
remove release notes and add comments
MOmarMiraj Apr 3, 2025
2474110
Merge branch 'main' into omar/automate-wheel-build
MOmarMiraj Apr 3, 2025
5750b30
remove unnecessary c omments
MOmarMiraj Apr 3, 2025
95c3aee
update setting up git user to use person who commited to be the one w…
MOmarMiraj Apr 4, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .github/workflows/prep-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: Prep Release

on:
workflow_dispatch:
inputs:
version:
description: "Version number:"
required: true
type: string
build_number:
description: "Build number:"
required: true
type: string

jobs:
prepare-release:
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/heads/sdk-core/') # Only run on branches that start with sdk-core/
steps:
- name: Checkout the code
uses: actions/checkout@v4

- name: Import GPG key
uses: crazy-max/ghaction-import-gpg@v6
with:
gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
git_user_signingkey: true
git_commit_gpgsign: true
git_tag_gpgsign: true
Comment on lines +26 to +29
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whose GPG key is this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be the private key of the bot that we will add to the secrets.


- name: Setup Git User
run: |
git config user.name "$GITHUB_ACTOR"
git config user.email "[email protected]"

- name: Parse and Validate Inputs
id: get_inputs
run: |
# Get inputs passed to the workflow
VERSION="${{ github.event.inputs.version }}"
BUILD_NUMBER="${{ github.event.inputs.build_number }}"

# Save the parsed values for future steps
echo "VERSION=$VERSION" >> $GITHUB_ENV
echo "BUILD_NUMBER=$BUILD_NUMBER" >> $GITHUB_ENV
shell: bash

- name: Run the Prep Release Script
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
make prep-release VERSION="$VERSION" BUILD_NUMBER="$BUILD_NUMBER"
shell: bash
29 changes: 29 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: Release SDK

on:
workflow_dispatch:

jobs:
Release-SDK:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit]

I think a better common pattern for job names is kebab-case:

Suggested change
Release-SDK:
release-sdk:

runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/heads/sdk-core/') # Only run on branches that start with sdk-core/
steps:
- name: Checkout the code
uses: actions/checkout@v4
- name: Import GPG key
uses: crazy-max/ghaction-import-gpg@v6
with:
gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
git_user_signingkey: true
git_commit_gpgsign: true
git_tag_gpgsign: true
- name: Setup Git User
run: |
git config user.name "$GITHUB_ACTOR"
git config user.email "[email protected]"

- name: Run the Release Script
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: make release
shell: bash
101 changes: 101 additions & 0 deletions .github/workflows/wheels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
name: Wheels Builder and Publisher
on:
pull_request:
branches:
- main
types:
- closed

jobs:
build_wheels:
name: Build wheels for Python SDK on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
if: github.event.pull_request.merged == true && contains(github.event.pull_request.head.ref, 'sdk-core/')
strategy:
fail-fast: false
matrix:
# macOS 13 is an Intel runner and macOS 14 is an Apple Silicon runner
os: [ubuntu-22.04, ubuntu-22.04-arm, windows-latest, macos-13, macos-14]
steps:
- uses: actions/checkout@v4
- name: Upgrade build dependencies
run: python -m pip install --upgrade pip setuptools wheel


# Need to grab the SDK version for the wheel name
- name: Extract SDK Version
Comment on lines +22 to +26
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit]

You can remove one extra line:

Suggested change
run: python -m pip install --upgrade pip setuptools wheel
# Need to grab the SDK version for the wheel name
- name: Extract SDK Version
run: python -m pip install --upgrade pip setuptools wheel
# Need to grab the SDK version for the wheel name
- name: Extract SDK Version

run: echo "SDK_VERSION=$(cat version.txt)" >> "$GITHUB_ENV"
shell: bash

- name: Install cibuildwheel
run: |
python -m pip install cibuildwheel

- name: Build wheels
env:
CIBW_SKIP: pp* *-musllinux_*
CIBW_MANYLINUX_X86_64_IMAGE: "quay.io/pypa/manylinux_2_34_x86_64"
CIBW_MANYLINUX_AARCH64_IMAGE: "quay.io/pypa/manylinux_2_34_aarch64"
CIBW_ARCHS: "native" # Equivalent to python's platform.machine()
CIBW_BEFORE_BUILD_WINDOWS: "pip install delvewheel"
CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: "delvewheel repair -w {dest_dir} {wheel}"
CIBW_TEST_REQUIRES: "pydantic pytest pytest-asyncio"
MACOSX_DEPLOYMENT_TARGET: "12.0"
CIBW_TEST_COMMAND: "python -m pytest {project}/src/onepassword/test_client.py"
OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.TEST_SERVICE_ACCOUNT_TOKEN }}
CIBW_ENVIRONMENT_PASS_LINUX: OP_SERVICE_ACCOUNT_TOKEN # We have to specify this to pass the token to the test command
run: |
python -m cibuildwheel --output-dir dist

- uses: actions/upload-artifact@v4
with:
name: onepassword-sdk-${{ env.SDK_VERSION }}-${{ matrix.os }}
path: ./dist/*.whl

build-sdist:
name: Build source distribution for Python SDK
runs-on: ubuntu-latest
if: github.event.pull_request.merged == true && contains(github.event.pull_request.head.ref, 'sdk-core/')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit]

Do we want to replace contains with startsWith since that reflects better what we're looking for?

steps:
- uses: actions/checkout@v4

- name: Extract SDK Version
run: echo "SDK_VERSION=$(cat version.txt)" >> "$GITHUB_ENV"
shell: bash

- name: Install dependencies
run: pip3 install build pydantic pytest pytest-asyncio

- name: Build source distribution
run: python3 -m build --sdist

- name: Test Source Distribution
env:
OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.TEST_SERVICE_ACCOUNT_TOKEN }}
run: |
python3 -m pip install dist/*.tar.gz
python3 -m pytest src/onepassword/test_client.py

- uses: actions/upload-artifact@v4
with:
name: onepassword-sdk-${{ env.SDK_VERSION }}
path: ./dist/*.tar.gz

publish-to-pypi:
name: Publish to PyPI
runs-on: ubuntu-latest
if: github.event.pull_request.merged == true && contains(github.event.pull_request.head.ref, 'sdk-core/')
environment:
name: pypi
url: https://pypi.org/project/onepassword-sdk/
permissions:
id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
needs: [build_wheels, build-sdist]
steps:
- uses: actions/download-artifact@v4
with:
pattern: onepassword-sdk-*
path: ./dist
merge-multiple: true
- name: Publish package distributions to PyPi
uses: pypa/gh-action-pypi-publish@release/v1.12
33 changes: 33 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
[build-system]
requires = ["setuptools>=66", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "onepassword-sdk"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth adding the version here. We do that for Connect SDK for Python.

dynamic = ["version"]
description = "The 1Password Python SDK offers programmatic read access to your secrets in 1Password in an interface native to Python."
authors = [{ name = "1Password" }]
license = { file = "LICENSE" }
readme = "README.md"
requires-python = ">=3.9"
classifiers = [
"Development Status :: 5 - Production/Stable",
"Operating System :: MacOS",
"Operating System :: POSIX :: Linux",
"Operating System :: Microsoft :: Windows",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"License :: OSI Approved :: MIT License",
]
dependencies = [
"pydantic>=2.5",
]

[project.urls]
Homepage = "https://github.com/1Password/onepassword-sdk-python"

[tool.setuptools.dynamic]
version = {file = "./version.txt"}
33 changes: 5 additions & 28 deletions setup.py
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of questions here:

  • Why did we choose to move to usinge pyproject.toml?
  • A lot of this setup part has been moved to pyproject.toml, right? If so, is there anything that should still live here even though we moved things?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyproject.toml is the standard and when we created the Python SDK... we used setuptools which is considered deprecated.

pyproject.toml doesn't have all the functionality that setuptools such as injecting custom functions like figuring out which shared library to include and other small things.

Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
from pathlib import Path
from setuptools import setup, find_packages
from setuptools import setup, find_packages, Distribution
from sysconfig import get_platform
from version import SDK_VERSION
import platform
import os

@@ -19,6 +17,9 @@ def finalize_options(self):
except ImportError:
bdist_wheel = None

class BinaryDistribution(Distribution):
def has_ext_modules(self):
return True

def get_shared_library_data_to_include():
# Return the correct uniffi C shared library extension for the given platform
@@ -46,35 +47,11 @@ def get_shared_library_data_to_include():


setup(
name="onepassword-sdk",
version=SDK_VERSION,
author="1Password",
long_description=(Path(__file__).parent / "README.md").read_text(),
long_description_content_type="text/markdown",
description="The 1Password Python SDK offers programmatic read access to your secrets in 1Password in an interface native to Python.",
url="https://github.com/1Password/onepassword-sdk-python",
packages=find_packages(
where="src",
),
license="MIT",
license_files="LICENSE",
distclass=BinaryDistribution,
package_dir={"": "src"},
python_requires=">=3.9",
classifiers=[
"Development Status :: 5 - Production/Stable",
"Operating System :: MacOS",
"Operating System :: POSIX :: Linux",
"Operating System :: Microsoft :: Windows",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"License :: OSI Approved :: MIT License",
],
cmdclass={"bdist_wheel": bdist_wheel},
package_data={"": get_shared_library_data_to_include()},
install_requires=[
"pydantic>=2.5", # Minimum Pydantic version to run the Python SDK
],
)
4 changes: 1 addition & 3 deletions src/release/scripts/build-wheels.sh
Original file line number Diff line number Diff line change
@@ -2,8 +2,6 @@

# Helper script to build the required wheels for the Python SDK

output_version_file="version.py"

# The list of python verisons the SDKs release for
python_versions=("$@")

@@ -15,7 +13,7 @@ macOS_version_x86_64=10.9
macOS_version_arm64=11.0

# Extracts the current verison number for cleanup function
current_version=$(awk -F "['\"]" '/SDK_VERSION =/{print $2}' "$output_version_file")
current_version=$(cat version.txt)

# Function to execute upon exit
cleanup() {
11 changes: 5 additions & 6 deletions src/release/scripts/prep-release.sh
Original file line number Diff line number Diff line change
@@ -2,21 +2,20 @@

# Helper script to prepare a release for the Python SDK.

output_version_file="version.py"
output_version_file="version.txt"
output_build_file="src/onepassword/build_number.py"
version_template_file="src/release/templates/version.tpl.py"
build_number_template_file="src/release/templates/build_number.tpl.py"


# Extracts the current build/version number for comparison and backup
current_version=$(awk -F "['\"]" '/SDK_VERSION =/{print $2}' "$output_version_file")
current_version=$(cat "$output_version_file")
current_build=$(awk -F "['\"]" '/SDK_BUILD_NUMBER =/{print $2}' "$output_build_file")

# Function to execute upon exit
cleanup() {
echo "Performing cleanup tasks..."
# Revert changes to file if any
sed -e "s/{{ version }}/$current_version/" "$version_template_file" > "$output_version_file"
echo -n "$current_version" > "$output_version_file"
sed -e "s/{{ build }}/$current_build/" "$build_number_template_file" > "$output_build_file"
exit 1
}
@@ -86,8 +85,8 @@ update_and_validate_version
# Update and validate the build number
update_and_validate_build

# Update version & build number in version.py and build_number.py respectively
sed -e "s/{{ version }}/$version/" "$version_template_file" > "$output_version_file"
# Update version & build number in version.txt and build_number.py respectively
echo -n "$version" > "$output_version_file"
sed -e "s/{{ build }}/$build/" "$build_number_template_file" > "$output_build_file"


9 changes: 1 addition & 8 deletions src/release/scripts/release.sh
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
set -e

# Read the contents of the files into variables
version=$(awk -F "['\"]" '/SDK_VERSION =/{print $2}' "version.py")
version=$(cat "version.txt")
build=$(awk -F "['\"]" '/SDK_BUILD_NUMBER =/{print $2}' "src/onepassword/build_number.py")
release_notes=$(< src/release/RELEASE-NOTES)

@@ -27,10 +27,3 @@ git tag -a -s "v${version}" -m "${version}"
git push origin tag "v${version}"

gh release create "v${version}" --title "Release ${version}" --notes "${release_notes}" --repo github.com/1Password/onepassword-sdk-python

# Release on PyPi
python3 -m twine upload dist/*

# Delete the dist folder after published
rm -r dist src/*.egg-info

1 change: 0 additions & 1 deletion src/release/templates/version.tpl.py

This file was deleted.

1 change: 0 additions & 1 deletion version.py

This file was deleted.

1 change: 1 addition & 0 deletions version.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.2.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need a version.txt file now? This feels a bit odd. If you want, it might be better to fetch the version from pyproject.toml file. If that's not possible, then better filenames can be version.py or if that doesn't work, .VERSION (check Kubernetes operator for an example).

Loading