Modernize python packaging #851

ddelange · 2025-03-06T20:34:48Z

Please pick a concise, informative and complete title for your PR.

The title is important because it will appear in our change log.

Motivation

Fix #859

Release is failing.

move to trusted publishing in release.yml
move to GitHub Release trigger in release.yml (good security for trusted publishing: git release permission vs general push permission)
add a dependabot.yml to check for github actions updates monthly (opens PRs to bump actions/setup-python etc if available)
deprecate setup.py and version.py in favour of pyproject.toml and setuptools_scm for git-based versioning
- consequently moved tests out of the smart_open folder using rename commit to retain blame (as we don't want tests in the whl/tar.gz)
- you can keep using v7.... as git tags prefix, it will be automatically stripped by setuptools_scm at runtime.
- no more need to push version bump commits. just release your Github Release and let it publish with that new tag you just created
Simplify release workflow README.md

a small demo of the versioning scheme of setuptools_scm:

$ git status
On branch pypa-publish
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   .github/workflows/release.yml
    modified:   pyproject.toml
    deleted:    setup.py
    modified:   smart_open/__init__.py
    deleted:    smart_open/version.py

$ git lg
* c299079 (HEAD -> pypa-publish, upstream/develop, origin/develop, origin/HEAD, develop) bump version to 7.2.0.dev0
*   e271ce6 Merge branch 'master' into develop
|\
| *   a08f0a0 (tag: v7.2.0, upstream/master) Merge branch 'release-7.2.0'
...

$ pip install -qe .
$ python -c 'import smart_open; print(smart_open.__version__)'
6.4.1.dev83+gc299079.d20250306

$ git pull upstream --tags
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 18 (delta 7), reused 9 (delta 7), pack-reused 8 (from 2)
Unpacking objects: 100% (18/18), 15.92 KiB | 2.27 MiB/s, done.
From https://github.com/RaRe-Technologies/smart_open
 * [new branch]      fix-ssh    -> upstream/fix-ssh
   299ca52..a08f0a0  master     -> upstream/master
 * [new tag]         v7.0.0     -> v7.0.0
 * [new tag]         v7.0.1     -> v7.0.1
 * [new tag]         v7.0.2     -> v7.0.2
 * [new tag]         v7.0.3     -> v7.0.3
 * [new tag]         v7.0.4     -> v7.0.4
 * [new tag]         v7.0.5     -> v7.0.5
 * [new tag]         v7.1.0     -> v7.1.0
 * [new tag]         v7.2.0     -> v7.2.0
You asked to pull from the remote 'upstream', but did not specify
a branch. Because this is not the default configured remote
for your current branch, you must specify a branch on the command line.

$ pip install -qe .
$ python -c 'import smart_open; print(smart_open.__version__)'
7.2.1.dev2+gc299079.d20250306

$ cd ~ && pip install -q https://github.com/ddelange/smart_open/releases/download/v7.2.1.rc6/smart_open-7.2.1rc6-py3-none-any.whl

$ python -c 'import smart_open; print(smart_open.__version__)'
7.2.1rc6

Please explain the motivation behind this PR.

If you're fixing a bug, link to the issue using a supported keyword like "Fixes #{issue_number}".

If you're adding a new feature, then consider opening a ticket and discussing it with the maintainers before you actually do the hard work.

Tests

Only delivery plumbing.

If you're fixing a bug, consider test-driven development:

Create a unit test that demonstrates the bug. The test should fail.

Implement your bug fix.

The test you created should now pass.

If you're implementing a new feature, include unit tests for it.

Make sure all existing unit tests pass.
You can run them locally using:
pytest smart_open
If there are any failures, please fix them before creating the PR (or mark it as WIP, see below).

Work in progress

If you're still working on your PR, mark the PR as draft PR.

We'll skip reviewing it for the time being.

Once it's ready, mark the PR as "ready for review", and ping one of the maintainers (e.g. mpenkov).

Checklist

Before you mark the PR as "ready for review", please make sure you have:

Picked a concise, informative and complete title
Clearly explained the motivation behind the PR
Linked to any existing issues that your PR will be solving
Included tests for any new functionality
Checked that all unit tests pass

Workflow

Please avoid rebasing and force-pushing to the branch of the PR once a review is in progress.

Rebasing can make your commits look a bit cleaner, but it also makes life more difficult from the reviewer, because they are no longer able to distinguish between code that has already been reviewed, and unreviewed code.

ddelange · 2025-03-06T20:37:36Z

@mpenkov apart from the TODO below, this should be good to go

@mpenkov can you please add piskvorky/smart_open plus release.yml as trusted publisher here?

ref trusted publishing

then I can do the plumbing in release.yml and I'll need to deprecate setup.py in favour of pyproject.toml. once that merges to master, you'll be unblocked

ref #848 (comment)

ddelange · 2025-03-06T21:35:30Z

release looks good on my fork, apart from that it fails because my fork is not a trusted publisher of course.

release.yml will only get triggered once for a release, namely when the Release is (pre)released. Edits to release description etc won't trigger the workflow.

mpenkov · 2025-03-08T11:29:47Z

Can you please update the release/ subdirectory? What's the new process for making a new release?

ddelange · 2025-03-08T17:46:28Z

release/README.md

+![New release](https://github.com/user-attachments/assets/cf8f2fa4-37c1-4e50-9fd8-ab6e3fd705b5)
+
+- Check that the [latest commit](https://github.com/piskvorky/smart_open/commits/develop) on `develop` passed all CI.
+- Make sure you're on `master` and you're up to date:
+    - `git checkout master && git pull`
+- Merge `develop` into `master`.
+    - `git pull origin develop --no-ff --no-edit && git push`
+- Draft a [new release](https://github.com/piskvorky/smart_open/releases/new).
+    - Fill in the new tag + enter.
+    - Confirm that it reads "Excellent! This tag will be created from the target when you publish this release.".
+    - Select target branch `master`.
+    - Click "Generate release notes" on the right top.
+    - Keep the tab open.
+- Copy the generated bullet points into `CHANGELOG.md`.
+- Commit `CHANGELOG.md` to `master` and push:
+  - `git checkout master && git pull && git add CHANGELOG.md && git commit -m "Update CHANGELOG.md" && git push`
+- Click "Publish release".
+- Github Actions `release.yml` is triggered, and uploads distributions to PyPI and to the Github Release.


@mpenkov how does this look? rendered version

multiple follow up commits, please look at the rendered version :)

I tried to reduce cognitive complexity as much as possible:

Added a new update_changelog.py which iterates over the diff and prepends a new section to CHANGELOG.md in the same format as usual.

The rest of the versioning is now fully automatic: 4 scripts less, no more copy pasting.

…open into pypa-publish * 'develop' of https://github.com/RaRe-Technologies/smart_open: Remove head_bucket call from s3.SinglepartWriter (piskvorky#857)

mpenkov · 2025-03-29T07:45:53Z

I'm a bit busy with my life outside of open source at the moment, but I'll have a look at this as soon as I get some spare cycles.

ddelange · 2025-03-29T07:51:21Z

release/update_help_txt.sh

deleted here as part of release folder clean up (wasn't and won't be used during release flow) and opened #853

* 'develop' of https://github.com/ddelange/smart_open: Add help.txt linting step (piskvorky#853)

ddelange · 2025-05-06T10:45:04Z

help.txt

-    tests (package)
    transport
    utils
-    version


first time #853 comes in handy: just pulled develop and lint started failing on this PR :)

mpenkov · 2025-05-18T09:53:08Z

release/README.md

+
+- Check that the [latest commit](https://github.com/piskvorky/smart_open/commits/develop) on `develop` passed all CI.
+- Make sure you're on `develop` and you're up to date locally:
+    - `git checkout develop && git pull`


Can we reduce the number of manual steps and copy/pasting here? Previously, releasing was a simple matter of calling prepare.sh and merge.sh and following the on-screen instructions, with occasional reference to this README.md.

fdd7e2e adds release/release.sh with chmod +x 👍

ddelange force-pushed the pypa-publish branch 9 times, most recently from 5dbee20 to 0b7b5af Compare March 6, 2025 21:34

ddelange force-pushed the pypa-publish branch from bd29a38 to 5aec5a3 Compare March 6, 2025 21:54

Modernize python packaging

66835f2

ddelange force-pushed the pypa-publish branch from 5aec5a3 to 66835f2 Compare March 6, 2025 21:55

Clean up release folder according to simplified release workflow

624b684

ddelange commented Mar 8, 2025

View reviewed changes

ddelange added 7 commits March 8, 2025 19:48

Typo

6159632

Clean up

2792f4f

Typo

4633b9d

Add links

a8ba147

Update docs to have CHANGELOG commits on develop

caa61a4

Delete annotate_pr.py

6be5afa

Add update_changelog.py

e760eba

ddelange mentioned this pull request Mar 14, 2025

use prefetch for sftp? #848

Closed

Merge branch 'develop' of https://github.com/RaRe-Technologies/smart_…

9cd101c

…open into pypa-publish * 'develop' of https://github.com/RaRe-Technologies/smart_open: Remove head_bucket call from s3.SinglepartWriter (piskvorky#857)

ddelange commented Mar 29, 2025

View reviewed changes

ddelange added 2 commits May 6, 2025 13:42

Merge branch 'develop' into pypa-publish

d2abfba

* 'develop' of https://github.com/ddelange/smart_open: Add help.txt linting step (piskvorky#853)

Run python update_helptext.py

b594a5f

ddelange commented May 6, 2025

View reviewed changes

ddelange mentioned this pull request May 7, 2025

Please release 7.2.0 to PyPI #859

Open

Add branch filters to push event

72daf9e

ddelange force-pushed the pypa-publish branch from d179be2 to 72daf9e Compare May 16, 2025 14:43

mpenkov reviewed May 18, 2025

View reviewed changes

ddelange added 2 commits May 18, 2025 16:06

Bring back release.sh

fdd7e2e

Do a local merge for cleaner commit message

31965b2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Modernize python packaging #851

Modernize python packaging #851

Uh oh!

ddelange commented Mar 6, 2025 •

edited

Loading

Uh oh!

ddelange commented Mar 6, 2025

Uh oh!

ddelange commented Mar 6, 2025 •

edited

Loading

Uh oh!

mpenkov commented Mar 8, 2025

Uh oh!

ddelange Mar 8, 2025

Uh oh!

ddelange Mar 8, 2025 •

edited

Loading

Uh oh!

ddelange Mar 8, 2025 •

edited

Loading

Uh oh!

mpenkov commented Mar 29, 2025

Uh oh!

ddelange Mar 29, 2025 •

edited

Loading

Uh oh!

ddelange May 6, 2025

Uh oh!

mpenkov May 18, 2025

Uh oh!

ddelange May 18, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Modernize python packaging #851

Are you sure you want to change the base?

Modernize python packaging #851

Uh oh!

Conversation

ddelange commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Tests

Work in progress

Checklist

Workflow

Uh oh!

ddelange commented Mar 6, 2025

Uh oh!

ddelange commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mpenkov commented Mar 8, 2025

Uh oh!

ddelange Mar 8, 2025

Choose a reason for hiding this comment

Uh oh!

ddelange Mar 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ddelange Mar 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpenkov commented Mar 29, 2025

Uh oh!

ddelange Mar 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ddelange May 6, 2025

Choose a reason for hiding this comment

Uh oh!

mpenkov May 18, 2025

Choose a reason for hiding this comment

Uh oh!

ddelange May 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ddelange commented Mar 6, 2025 •

edited

Loading

ddelange commented Mar 6, 2025 •

edited

Loading

ddelange Mar 8, 2025 •

edited

Loading

ddelange Mar 8, 2025 •

edited

Loading

ddelange Mar 29, 2025 •

edited

Loading

ddelange May 18, 2025 •

edited

Loading