Skip to content

Commit cc1ab2c

Browse files
kevinjqliukevinjqliu
authored
Improve documentation for "how to release" (#1359)
* initial update * edits * add gpg instructions * verify artifacts * add twine not * grammar * edits * remove old artifacts * update doc workflow action * and name * add docs on patch vs major/minor release
1 parent c21aefd commit cc1ab2c

File tree

1 file changed

+180
-58
lines changed

1 file changed

+180
-58
lines changed

‎mkdocs/docs/how-to-release.md‎

Lines changed: 180 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,31 @@
1717
- under the License.
1818
-->
1919

20-
# How to release
20+
# How to Release
2121

22-
The guide to release PyIceberg.
22+
This guide outlines the process for releasing PyIceberg in accordance with the [Apache Release Process](https://infra.apache.org/release-publishing.html). The steps include:
2323

24-
The first step is to publish a release candidate (RC) and publish it to the public for testing and validation. Once the vote has passed on the RC, the RC turns into the new release.
24+
1. Preparing for a release
25+
2. Publishing a Release Candidate (RC)
26+
3. Community Voting and Validation
27+
4. Publishing the Final Release (if the vote passes)
28+
5. Post-Release Step
2529

26-
## Preparing for a release
30+
## Requirements
2731

28-
Before running the release candidate, we want to remove any APIs that were marked for removal under the @deprecated tag for this release.
32+
* A GPG key must be registered and published in the [Apache Iceberg KEYS file](https://downloads.apache.org/iceberg/KEYS). Follow [the instructions for setting up a GPG key and uploading it to the KEYS file](#set-up-gpg-key-and-upload-to-apache-iceberg-keys-file).
33+
* SVN Access
34+
* Permission to upload artifacts to the [Apache development distribution](https://dist.apache.org/repos/dist/dev/iceberg/) (requires Apache Commmitter access).
35+
* Permission to upload artifacts to the [Apache release distribution](https://dist.apache.org/repos/dist/release/iceberg/) (requires Apache PMC access).
36+
* PyPI Access
37+
* The `twine` package must be installed for uploading releases to PyPi.
38+
* A PyPI account with publishing permissions for the [pyiceberg project](https://pypi.org/project/pyiceberg/).
39+
40+
## Preparing for a Release
41+
42+
### Remove Deprecated APIs
43+
44+
Before running the release candidate, we want to remove any APIs that were marked for removal under the `@deprecated` tag for this release. See [#1269](https://github.com/apache/iceberg-python/pull/1269).
2945

3046
For example, the API with the following deprecation tag should be removed when preparing for the 0.2.0 release.
3147

@@ -48,23 +64,46 @@ deprecation_message(
4864
)
4965
```
5066

51-
## Running a release candidate
67+
### Update Library Version
68+
69+
Update the version in `pyproject.toml` and `pyiceberg/__init__.py` to match the release version. See [#1276](https://github.com/apache/iceberg-python/pull/1276).
70+
71+
## Publishing a Release Candidate (RC)
72+
73+
### Release Types
74+
75+
#### Major/Minor Release
5276

53-
Make sure that the version is correct in `pyproject.toml` and `pyiceberg/__init__.py`. Correct means that it reflects the version that you want to release.
77+
* Use the `main` branch for the release.
78+
* Includes new features, enhancements, and any necessary backward-compatible changes.
79+
* Examples: `0.8.0`, `0.9.0`, `1.0.0`.
5480

55-
### Setting the tag
81+
#### Patch Release
5682

57-
Make sure that you're on the right branch, and the latest branch:
83+
* Use the branch corresponding to the patch version, such as `pyiceberg-0.8.x`.
84+
* Focuses on critical bug fixes or security patches that maintain backward compatibility.
85+
* Examples: `0.8.1`, `0.8.2`.
5886

59-
For a Major/Minor release, make sure that you're on `main`, for patch versions the branch corresponding to the version that you want to patch, i.e. `pyiceberg-0.6.x`.
87+
To create a patch branch from the latest release tag:
6088

6189
```bash
62-
git checkout <branch>
63-
git fetch --all
64-
git reset --hard apache/<branch>
90+
# Check out the base branch for the patch version
91+
git checkout pyiceberg-0.8.x
92+
93+
# Create a new branch for the upcoming patch release
94+
git checkout -b pyiceberg-0.8.1
6595
```
6696

67-
Set the tag on the last commit:
97+
### Create Tag
98+
99+
Ensure you are on the correct branch:
100+
101+
* For a major/minor release, use the `main` branch
102+
* For a patch release, use the branch corresponding to the patch version, i.e. `pyiceberg-0.6.x`.
103+
104+
Create a signed tag:
105+
106+
Replace `VERSION` and `RC` with the appropriate values for the release.
68107

69108
```bash
70109
export RC=rc1
@@ -74,48 +113,49 @@ export VERSION_BRANCH=${VERSION_WITHOUT_RC//./-}
74113
export GIT_TAG=pyiceberg-${VERSION}
75114

76115
git tag -s ${GIT_TAG} -m "PyIceberg ${VERSION}"
77-
git push apache ${GIT_TAG}
78-
79-
export GIT_TAG_REF=$(git show-ref ${GIT_TAG})
80-
export GIT_TAG_HASH=${GIT_TAG_REF:0:40}
81-
export LAST_COMMIT_ID=$(git rev-list ${GIT_TAG} 2> /dev/null | head -n 1)
116+
git push [email protected]:apache/iceberg-python.git ${GIT_TAG}
82117
```
83118

84-
The `-s` option will sign the commit. If you don't have a key yet, you can find the instructions [here](http://www.apache.org/dev/openpgp.html#key-gen-generate-key). To install gpg on a M1 based Mac, a couple of additional steps are required: <https://gist.github.com/phortuin/cf24b1cca3258720c71ad42977e1ba57>.
85-
If you have not published your GPG key in [KEYS](https://downloads.apache.org/iceberg/KEYS) yet, you must publish it before sending the vote email by doing:
86-
87-
```bash
88-
svn co https://dist.apache.org/repos/dist/release/iceberg icebergsvn
89-
cd icebergsvn
90-
echo "" >> KEYS # append a newline
91-
gpg --list-sigs <YOUR KEY ID HERE> >> KEYS # append signatures
92-
gpg --armor --export <YOUR KEY ID HERE> >> KEYS # append public key block
93-
svn commit -m "add key for <YOUR NAME HERE>"
94-
```
119+
### Publish Release Candidate (RC)
95120

96-
### Upload to Apache SVN
121+
#### Upload to Apache Dev SVN
97122

98-
Both the source distribution (`sdist`) and the binary distributions (`wheels`) need to be published for the RC. The wheels are convenient to avoid having people to install compilers locally. The downside is that each architecture requires its own wheel. [use `cibuildwheel`](https://github.com/pypa/cibuildwheel) runs in Github actions to create a wheel for each of the architectures.
123+
##### Create Artifacts for SVN
99124

100-
Before committing the files to the Apache SVN artifact distribution SVN hashes need to be generated, and those need to be signed with gpg to make sure that they are authentic.
125+
Run the [`Python release` Github Action](https://github.com/apache/iceberg-python/actions/workflows/python-release.yml).
101126

102-
Go to [Github Actions and run the `Python release` action](https://github.com/apache/iceberg-python/actions/workflows/python-release.yml). **Set the version to main, since we cannot modify the source**.
127+
* Tag: Use the newly created tag.
128+
* Version: Set the `version` to `main`, as the source cannot be modified.
103129

104130
![Github Actions Run Workflow for SVN Upload](assets/images/ghactions-run-workflow-svn-upload.png)
105131

106-
Download the zip, and sign the files:
132+
This action will generate:
133+
134+
* Source distribution (`sdist`)
135+
* Binary distributions (`wheels`) for each architectures. These are created using [`cibuildwheel`](https://github.com/pypa/cibuildwheel)
136+
137+
##### Download Artifacts, Sign, and Generate Checksums
138+
139+
Download the ZIP file containing the artifacts from the GitHub Actions run and unzip it.
140+
141+
Navigate to the release directory. Sign the files and generate checksums:
142+
143+
* `.asc` files: GPG-signed versions of each artifact to ensure authenticity.
144+
* `.sha512` files: SHA-512 checksums for verifying file integrity.
107145

108146
```bash
109147
cd release-main/
110148

111149
for name in $(ls pyiceberg-*.whl pyiceberg-*.tar.gz)
112150
do
113-
gpg --yes --armor --local-user [email protected] --output "${name}.asc" --detach-sig "${name}"
151+
gpg --yes --armor --output "${name}.asc" --detach-sig "${name}"
114152
shasum -a 512 "${name}" > "${name}.sha512"
115153
done
116154
```
117155

118-
Now we can upload the files from the same directory:
156+
##### Upload Artifacts to Apache Dev SVN
157+
158+
Now, upload the files from the same directory:
119159

120160
```bash
121161
export SVN_TMP_DIR=/tmp/iceberg-${VERSION_BRANCH}/
@@ -128,21 +168,59 @@ svn add $SVN_TMP_DIR_VERSIONED
128168
svn ci -m "PyIceberg ${VERSION}" ${SVN_TMP_DIR_VERSIONED}
129169
```
130170

131-
### Upload to PyPi
171+
Verify the artifact is uploaded to [https://dist.apache.org/repos/dist/dev/iceberg](https://dist.apache.org/repos/dist/dev/iceberg/).
172+
173+
##### Remove Old Artifacts From Apache Dev SVN
174+
175+
Clean up old RC artifacts:
176+
177+
```bash
178+
svn delete https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-<OLD_RC_VERSION> -m "Remove old RC artifacts"
179+
```
180+
181+
#### Upload to PyPi
182+
183+
##### Create Artifacts for PyPi
132184

133-
Go to Github Actions and run the `Python release` action again. This time, set the **version** of the release candidate as the input: e.g. `0.7.0rc1`. Download the zip and unzip it locally.
185+
Run the [`Python release` Github Action](https://github.com/apache/iceberg-python/actions/workflows/python-release.yml).
186+
187+
* Tag: Use the newly created tag.
188+
* Version: Set the `version` to release candidate, e.g. `0.7.0rc1`.
134189

135190
![Github Actions Run Workflow for PyPi Upload](assets/images/ghactions-run-workflow-pypi-upload.png)
136191

137-
Next step is to upload them to pypi. Please keep in mind that this **won't** bump the version for everyone that hasn't pinned their version, since it is set to an RC [pre-release and those are ignored](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#pre-release-versioning).
192+
##### Download Artifacts
193+
194+
Download the zip file from the Github Action run and unzip locally.
195+
196+
##### Upload Artifacts to PyPi
197+
198+
Upload release candidate to PyPi. This **won't** bump the version for everyone that hasn't pinned their version, since it is set to an RC [pre-release and those are ignored](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#pre-release-versioning).
199+
200+
<!-- prettier-ignore-start -->
201+
202+
!!! note
203+
`twine` might require an PyPi API token.
204+
205+
<!-- prettier-ignore-end -->
138206

139207
```bash
140-
twine upload release-0.7.0rc1/*
208+
twine upload release-${VERSION}/*
141209
```
142210

211+
Verify the artifact is uploaded to [PyPi](https://pypi.org/project/pyiceberg/#history).
212+
213+
## Vote
214+
215+
### Generate Vote Email
216+
143217
Final step is to generate the email to the dev mail list:
144218

145219
```bash
220+
export GIT_TAG_REF=$(git show-ref ${GIT_TAG})
221+
export GIT_TAG_HASH=${GIT_TAG_REF:0:40}
222+
export LAST_COMMIT_ID=$(git rev-list ${GIT_TAG} 2> /dev/null | head -n 1)
223+
146224
cat << EOF > release-announcement-email.txt
147225
To: [email protected]
148226
Subject: [VOTE] Release Apache PyIceberg $VERSION
@@ -185,12 +263,19 @@ Please vote in the next 72 hours.
185263
[ ] +0
186264
[ ] -1 Do not release this because...
187265
EOF
188-
189-
cat release-announcement-email.txt
190266
```
191267

192-
## Vote has passed
268+
### Send Vote Email
269+
270+
Verify the content of `release-announcement-email.txt` and send it to `[email protected]` with the corresponding subject line.
193271

272+
## Vote has failed
273+
274+
If there are concerns with the RC, address the issues and generate another RC.
275+
276+
## Publish the Final Release (Vote has passed)
277+
278+
A minimum of 3 binding +1 votes is required to pass an RC.
194279
Once the vote has been passed, you can close the vote thread by concluding it:
195280

196281
```text
@@ -205,36 +290,54 @@ The release candidate has been accepted as PyIceberg <VERSION>. Thanks everyone,
205290
Kind regards,
206291
```
207292

208-
### Copy the artifacts to the release dist
293+
### Upload the accepted RC to Apache Release SVN
294+
<!-- prettier-ignore-start -->
209295

210-
```bash
211-
export RC=rc2
212-
export VERSION=0.7.0${RC}
213-
export VERSION_WITHOUT_RC=${VERSION/rc?/}
296+
!!! note
297+
Only a PMC member has the permission to upload an artifact to the SVN release dist.
214298

299+
<!-- prettier-ignore-end -->
300+
301+
```bash
215302
export SVN_DEV_DIR_VERSIONED="https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-${VERSION}"
216303
export SVN_RELEASE_DIR_VERSIONED="https://dist.apache.org/repos/dist/release/iceberg/pyiceberg-${VERSION_WITHOUT_RC}"
217304

218305
svn mv ${SVN_DEV_DIR_VERSIONED} ${SVN_RELEASE_DIR_VERSIONED} -m "PyIceberg: Add release ${VERSION_WITHOUT_RC}"
219306
```
220307

221-
<!-- prettier-ignore-start -->
308+
Verify the artifact is uploaded to [https://dist.apache.org/repos/dist/release/iceberg](https://dist.apache.org/repos/dist/release/iceberg/).
222309

223-
!!! note
224-
Only a PMC member has the permission to upload an artifact to the SVN release dist.
310+
### Remove Old Artifacts From Apache Release SVN
225311

226-
<!-- prettier-ignore-end -->
312+
We only want to host the latest release. Clean up old release artifacts:
313+
314+
```bash
315+
svn delete https://dist.apache.org/repos/dist/release/iceberg/pyiceberg-<OLD_RELEASE_VERSION> -m "Remove old release artifacts"
316+
```
227317

228318
### Upload the accepted release to PyPi
229319

230320
The latest version can be pushed to PyPi. Check out the Apache SVN and make sure to publish the right version with `twine`:
231321

322+
<!-- prettier-ignore-start -->
323+
324+
!!! note
325+
`twine` might require an PyPi API token.
326+
327+
<!-- prettier-ignore-end -->
328+
232329
```bash
233330
svn checkout https://dist.apache.org/repos/dist/release/iceberg /tmp/iceberg-dist-release/
234331
cd /tmp/iceberg-dist-release/pyiceberg-${VERSION_WITHOUT_RC}
235332
twine upload pyiceberg-*.whl pyiceberg-*.tar.gz
236333
```
237334

335+
Verify the artifact is uploaded to [PyPi](https://pypi.org/project/pyiceberg/#history).
336+
337+
## Post Release
338+
339+
### Send out Release Announcement Email
340+
238341
Send out an announcement on the dev mail list:
239342

240343
```text
@@ -253,19 +356,19 @@ This Python release can be downloaded from: https://pypi.org/project/pyiceberg/<
253356
Thanks to everyone for contributing!
254357
```
255358

256-
## Release the docs
359+
### Release the docs
257360

258-
A committer triggers the [`Python Docs` Github Actions](https://github.com/apache/iceberg-python/actions/workflows/python-ci-docs.yml) through the UI by selecting the branch that just has been released. This will publish the new docs.
361+
Run the [`Release Docs` Github Action](https://github.com/apache/iceberg-python/actions/workflows/python-release-docs.yml).
259362

260-
## Update the Github template
363+
### Update the Github template
261364

262365
Make sure to create a PR to update the [GitHub issues template](https://github.com/apache/iceberg-python/blob/main/.github/ISSUE_TEMPLATE/iceberg_bug_report.yml) with the latest version.
263366

264-
## Update the integration tests
367+
### Update the integration tests
265368

266369
Ensure to update the `PYICEBERG_VERSION` in the [Dockerfile](https://github.com/apache/iceberg-python/blob/main/dev/Dockerfile).
267370

268-
## Create a Github Release Note
371+
### Create a Github Release Note
269372

270373
Create a [new Release Note](https://github.com/apache/iceberg-python/releases/new) on the iceberg-python Github repository.
271374

@@ -278,3 +381,22 @@ Then, select the previous release version as the **Previous tag** to use the dif
278381
**Generate release notes**.
279382

280383
**Set as the latest release** and **Publish**.
384+
385+
## Misc
386+
387+
### Set up GPG key and Upload to Apache Iceberg KEYS file
388+
389+
To set up GPG key locally, see the instructions [here](http://www.apache.org/dev/openpgp.html#key-gen-generate-key).
390+
391+
To install gpg on a M1 based Mac, a couple of additional steps are required: <https://gist.github.com/phortuin/cf24b1cca3258720c71ad42977e1ba57>.
392+
393+
Then, published GPG key to the [Apache Iceberg KEYS file](https://downloads.apache.org/iceberg/KEYS):
394+
395+
```bash
396+
svn co https://dist.apache.org/repos/dist/release/iceberg icebergsvn
397+
cd icebergsvn
398+
echo "" >> KEYS # append a newline
399+
gpg --list-sigs <YOUR KEY ID HERE> >> KEYS # append signatures
400+
gpg --armor --export <YOUR KEY ID HERE> >> KEYS # append public key block
401+
svn commit -m "add key for <YOUR NAME HERE>"
402+
```

0 commit comments

Comments
 (0)