This describes how to release Dagger:
- π Engine + π CLI β±
30mins - πΉ Go SDK β±
30mins - π Python SDK β±
5mins - β¬’ TypeScript SDK β±
5mins - π§ͺ Elixir SDK β±
5mins - π PHP SDK β±
5mins - βΈοΈ Helm chart β±
2mins - π Documentation β±
5mins - π Playground β±
2mins - π Daggerverse β±
2mins - βοΈ Dagger Cloud β±
2mins - πͺ£ Install scripts β±
2mins - π dagger-for-github β±
10mins - πΊ dagger Homebrew β±
2mins - βοΈ nix β±
2mins - βοΈ CI β±
2mins
This is a high-level diagram of how all the pieces fit together:
flowchart TB
repo(["π github.com/dagger/dagger"])
docs["π Documentation"]
ci["βοΈ CI"]
repo -.-> docs & ci
subgraph Dagger
engine("π Engine")
cli("π CLI ")
end
repo ==> engine & cli
S3["π dl.dagger.io/dagger"]
brew-tap["π github.com/dagger/homebrew-tap"]
github-action["π github.com/dagger/dagger-for-github"]
nix["βοΈ github.com/dagger/nix"]
cli --> S3 ------> brew-tap & github-action & nix
registry["π¦ registry.dagger.io/engine"]
ghcr["π ghcr.io/dagger/engine"]
engine --> ghcr --> registry
go["πΉ Go SDK"]
go-repo["π github.com/dagger/dagger-go-sdk"]
go-pkg["πΉ dagger.io/dagger"]
go-ref["πΉ pkg.go.dev/dagger.io/dagger"]
playground["π Playground"]
daggerverse["π Daggerverse"]
cloud["βοΈ Dagger Cloud"]
repo ==> go --> go-repo --> go-pkg & go-ref
go-pkg -.-> daggerverse & cloud
registry -.- S3 -.- go & python & typescript & elixir & php & helm
registry -.....- playground
python["π Python SDK"]
pypi["π pypi.org/project/dagger-io"]
readthedocs["π dagger-io.readthedocs.io"]
repo ==> python --> pypi & readthedocs
typescript["β¬’ TypeScript SDK"]
npm["β¬’ npmjs.com/@dagger.io/dagger"]
repo ==> typescript --> npm
elixir["π§ͺ Elixir SDK"]
hex["π§ͺ hex.pm/packages/dagger"]
repo ==> elixir --> hex
php["π PHP SDK"]
php-repo["π github.com/dagger/dagger-php-sdk"]
php-pkg["π packagist.org/packages/dagger/dagger"]
repo ======> php --> php-repo --> php-pkg
helm["βΈοΈ Helm chart"]
repo ======> helm
Before you go ahead and produce a new release, remember that it's a team effort. The first step is to let the team know what is going to happen, preferably a few days in advance so that they can react. We do our release threads in public wherever possible, to give the community more visibility, and to help external contributors to more easily keep in sync with the release cadence.
To let the team know:
- Create a new milestone in GitHub,
e.g.
v0.11.5 Milestone - Create a new post in Discord #engine-dev,
e.g.
v0.11.5 release - 28th May 2024
This allows others to weigh in whether:
- we should go for a patch / minor bump
- there are any PRs that people are waiting to get merged
- any big features which need to remain experimental?
- etc.
Maybe there are breaking changes which we should be aware of and message accordingly. Giving other team members a day or two to react - because timezones! - will make this entire process smoother.
Most importantly, patch vs minor is not a technical decision. If you want to read more about this, see this (private) Discord thread.
- If doing a minor release, determine if there are any deprecations that can be removed.
Note
Once you know what type of release we are producing - patch vs minor -
remember to edit the ? in the Discord thread.
Where possible, we try to ensure backwards compatibility between mismatched cli and engine versions. However, for technical reasons, this isn't always possible: sometime the communication protocol changes, or a bug fix or new feature requires changes on both the CLI and the engine.
Before releasing, make sure to sanity check the backwards compatibility of a release. If you enounter issues, then:
- Add a release note using
changie new(or add it later manually). - Bump the minimum version numbers in engine/version.go.
If unsure, bump both the client and engine minimum version numbers, but if the backwards compatibility is only an issue in one direction, you only need to bump that one.
In order to keep this relevant & accurate, we improve this doc during the release process. It's the best time to pause, observe how it all fits together, and improve it. We want small, constant improvements which compound. Therefore:
- Save a copy of this doc outside of this repository (e.g.
~/Downloads/RELEASING.md). Now open that copy in your editor and start ticking items off it as you make progress. Remember to add / remove / edit any parts which could be improved. As inspiration, see what a past PR with improvements looks like. - Update the date in the shields.io badge, first line in this file.
Note
We believe in documentation first, automation second. Documenting a process forces us to understand it well. Continuously editing this documentation refines our understanding. Once we have a good grasp of the problem space, and reach an elegant solution, it comes natural to automate & speed things up, to make the process more efficient. We should still be able to perform things manually if we need to - because sometimes automation fails π€·. This is when everyone wishes they had good documentation, the original author(s) or both! It's also worth mentioning that when it's time to improve this automation, we want to be looking at the blueprint - this doc right here - not the implementation. If you ever had to migrate from Chef/Puppet to Ansible/Terraform, you know what it was like to migrate the implementation.
Currently, some of the steps here vary depending on whether we are running a release off of main vs. off of a separate branch.
They are marked with π¨ in the steps below.
We do this process when we need to do a patch release but don't want to include every commit that has been merged to main since the last release.
In the long term we should strive to make the process the same regardless of the branch we are releasing off of, but for now, we need to be aware of the differences.
Warning
It is important to always do an Engine + CLI release prior to releasing any SDK. This will ensure that all the APIs in the SDK are also available in the Engine it depends on.
Note
If you do not have changie installed, see https://changie.dev
Setup some variables used throughout the release process:
# π¨ change this from `main` to `release-vX.Y.Z` if releasing off a non-main branch
export RELEASE_BRANCH=main
# If not named "origin" in your local checkout, replace "origin" with whatever the
# github.com/dagger/dagger repo is named for you locally
export DAGGER_REPO_REMOTE=originNote
This process involves direct pushes to the github.com/dagger/dagger repository. Recently there have been permission issues here that result in errors like
refusing to allow an OAuth App to create or update workflow when pushing.
If you encounter this, you may need to change your git remote from https to ssh, e.g.
git remote set-url $DAGGER_REPO_REMOTE git@github.com:dagger/dagger.git
It's advisable to go back to https after the release is complete to not retain excessive permissions longer than necessary, e.g. to go back run
git remote set-url $DAGGER_REPO_REMOTE https://github.com/dagger/dagger.git
π¨ Non-main branch release only:
Create a new branch off of the previous git tag being patched named release-vX.Y.Z (e.g. release-v0.11.9)
git switch -c ${RELEASE_BRANCH:?must be set}
# e.g. if creating a new v0.11.9 release off of v0.11.8, use v0.11.8 here
git reset --hard <previous tag being patched>
git push --set-upstream $DAGGER_REPO_REMOTE "${RELEASE_BRANCH:?must be set}"Setup the local branch to align with the remote branch being released
git checkout "${RELEASE_BRANCH:?must be set}"
git pull $DAGGER_REPO_REMOTE "${RELEASE_BRANCH:?must be set}"
export ENGINE_GIT_SHA="$(git rev-parse --verify HEAD)"π¨ Non-main branch release only:
git cherry-pick <commit-hash> the commits you want to include in the release from main to the release branch.
You will also want to ensure you always cherry-pick a few special commits:
- The SDK bump PR for the release being patched.
- e.g. if we are creating a
v0.11.9release based off ofv0.11.8, you want to ensure you cherry-pick this PR.
- e.g. if we are creating a
- The PR that bumps CI to use the new Engine version.
- e.g. if we are creating a
v0.11.9release based off ofv0.11.8, you want to ensure you cherry-pick this PR that upgrades CI to usev0.11.8engines.
- e.g. if we are creating a
-
Create e.g.
.changes/v0.11.9.mdby either runningchangie batch patch(orchangie batch minorif this is a new minor). -
Make any necessary edits to the newly generated file, e.g.
.changes/v0.11.9.md -
Update
CHANGELOG.mdby runningchangie merge. -
30 minsSubmit a PR - e.g.add-v0.11.9-release-noteswith the new release notes so that they can be used in the new release.- π¨ Non-main branch release only: This PR will also include the cherry-picked commits mentioned above.
-
Get the PR reviewed & merged. The merge commit is what gets tagged in the next step.
- π¨ Non-main branch release only: Ideally use "Rebase and Merge" rather than squashing commits when merging so we can more easily preserve the history of the cherry-picked commits.
-
Ensure that all checks are green β for the
<ENGINE_GIT_SHA>on the<RELEASE_BRANCH>that you are about to release.- π¨ Non-main branch release only: currently, CI does not run on non-main branches and some of the workflows are currently hardcoded with
mainso it's not safe to manually run them. So for now this has to be skipped in this case.
- π¨ Non-main branch release only: currently, CI does not run on non-main branches and some of the workflows are currently hardcoded with
-
30minsWhen you have confirmed that all checks are green, run the following:
export ENGINE_VERSION="$(changie latest)"
git tag "${ENGINE_VERSION:?must be set}" "${ENGINE_GIT_SHA:?must be set}"
git push "${DAGGER_REPO_REMOTE:?must be set}" "${ENGINE_VERSION:?must be set}"
# This is required to interpolate $ENGINE_VERSION to the SDK release notes
export CHANGIE_ENGINE_VERSION="$ENGINE_VERSION"This will kick off
.github./workflows/publish.yml.
After the publish job in this workflow passes, a new draft PR will
automatically be created to bump the Engine version in the various SDKs.
π¨ Non-main branch release only:
Currently, the bump-engine PR will be created against main by default and also include commits from main rather than just the ones you cherry-picked.
You will want to leave this PR open, but need to manually create a new PR against the release branch, with an additional cherry-pick on the engine bump commit.
βΌοΈ Be sure to change the branch being merged into frommainto the release branch when opening the PR.
This PR will also need the extra commits described below for SDK changelogs, helm chart version bump, etc.
An example PR used while releasing v0.11.9 off of v0.11.8 can be found here.
- Checkout the
bump-enginebranch locally & generate changelogs for all SDKs:
# Fill in the value with the PR number of the bump-engine PR just created
export BUMP_ENGINE_PR=git fetch "${DAGGER_REPO_REMOTE:?must be set}"
git checkout --track "${DAGGER_REPO_REMOTE:?must be set}/bump-engine"
cd sdk/go
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=${BUMP_ENGINE_PR:?must be set}"
changie batch patch
changie merge
cd ../python
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=${BUMP_ENGINE_PR:?must be set}"
changie batch patch
changie merge
cd ../typescript
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=${BUMP_ENGINE_PR:?must be set}"
changie batch patch
changie merge
cd ../elixir
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=${BUMP_ENGINE_PR:?must be set}"
changie batch patch
changie merge
cd ../php
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=${BUMP_ENGINE_PR:?must be set}"
changie batch patch
changie merge
cd ../../helm/dagger
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=${BUMP_ENGINE_PR:?must be set}"
changie batch patch
changie merge
cd ../..- Commit and push the changes with the message
Add SDK release notes - Update all dagger versions in
docs/current_docs/partials/_install-cli.mdxto$ENGINE_VERSION -
30minsOpen this draft PR in github.com/dagger/dagger/pulls & click on Ready to review. - After all checks pass, merge this PR. Tip: go to the Files changed tab on the PR to review without an explicit request.
- Ensure that all checks are green β
for the
<SDK_GIT_SHA>on the<RELEASE_BRANCH>branch that you are about to release. This will usually be the commit that bumps the Engine version, the one that you merged earlier.- π¨ Non-main branch release only: currently, CI does not run on non-main branches and some of the workflows are currently hardcoded with
mainso it's not safe to manually run them. So for now this has to be skipped in this case.
- π¨ Non-main branch release only: currently, CI does not run on non-main branches and some of the workflows are currently hardcoded with
git checkout "${RELEASE_BRANCH:?must be set}"
git pull "${DAGGER_REPO_REMOTE:?must be set}"
git branch -D bump-engine
export SDK_GIT_SHA="$(git rev-parse --verify HEAD)"- Tag & publish:
cd sdk/go && export GO_SDK_VERSION=$(changie latest) && cd ../..
git tag "sdk/go/${GO_SDK_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push "${DAGGER_REPO_REMOTE:?must be set}" "sdk/go/${GO_SDK_VERSION:?must be set}"This will trigger the publish-sdk-go
workflow
which publishes to π
github.com/dagger/dagger-go-sdk.
-
20minsBump the Go SDK version in our internal CI targets & check that Engine tests pass locally. If everything looks good, submit a new PR with this change so that we can check that all our workflows pass with the new SDK version before we create a new GitHub release and make it widely public.
go mod edit -require dagger.io/dagger@${GO_SDK_VERSION:?must be set}
go mod edit -require github.com/dagger/dagger/engine/distconsts@${GO_SDK_VERSION:?must be set}
go mod tidy
cd ci
go mod edit -require github.com/dagger/dagger/engine/distconsts@${ENGINE_VERSION:?must be set}
go mod tidy
cd ..-
Update all dagger versions in
.github/to$ENGINE_VERSION- The version numbers (of the form
<major>.<minor>.<patch>) should be updated to the new version - The worker runner versions (of the form
dagger-v<major>-<minor>-<patch>-<worker>)
- The version numbers (of the form
-
Open a PR with the title
Improve Releasing during $ENGINE_VERSION
git checkout -b improve-releasing-during-${ENGINE_VERSION:?must be set}
git add . # or any other files changed during the last few steps
git commit -s -m "Improve releasing during $ENGINE_VERSION"
git pushπ¨ Non-main branch release only:
Change the branch the PR is being merged into from main to the release-vX.Y.Z branch.
Ensure that all the workflows succeed before continuing (specifically test and testdev)!
- Download and install the latest release, and continue the rest of the release process using the just-released CLI.
curl -L https://dl.dagger.io/dagger/install.sh | BIN_DIR=$HOME/.local/bin DAGGER_VERSION=0.11.9 sh
# install the cli to dagger-0.11.9, and symlink dagger to it
mv ~/.local/bin/dagger{,-0.11.9}
ln -s ~/.local/bin/dagger{-0.11.9,}
dagger version- After you confirm that our internal tooling works with the new Go SDK
release, π github.com/dagger/dagger-go-sdk,
double-check that is was picked up by pkg.go.dev.
You can manually request this new version via
open https://pkg.go.dev/dagger.io/dagger@${GO_SDK_VERSION:?must be set}. The new version can take up to60minsto appear, it's OK to move on.
Note
To upload the release notes, we need to have the gh
CLI installed, e.g. brew install gh
- Upload the release notes by running:
gh release create "sdk/go/${GO_SDK_VERSION:?must be set}" \
--draft --verify-tag --title sdk/go/$GO_SDK_VERSION \
--notes-file sdk/go/.changes/$GO_SDK_VERSION.md- Check that release notes look good in
Preview -
β οΈ De-select Set as the latest release (only used for π Engine + π CLI releases) - Click on Publish release
- Tag & publish:
git checkout "${RELEASE_BRANCH:?must be set}"
cd sdk/python && export PYTHON_SDK_VERSION=$(changie latest) && cd ../..
git tag "sdk/python/${PYTHON_SDK_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push "${DAGGER_REPO_REMOTE:?must be set}" sdk/python/${PYTHON_SDK_VERSION}This will trigger the Publish Python SDK
workflow
which publishes dagger-io to π PyPI
- Upload the release notes by running:
gh release create "sdk/python/${PYTHON_SDK_VERSION:?must be set}" \
--draft --verify-tag --title sdk/python/$PYTHON_SDK_VERSION \
--notes-file sdk/python/.changes/$PYTHON_SDK_VERSION.md-
β οΈ De-select Set as the latest release (only used for π Engine + π CLI releases) - Check that release notes look good in
Preview. FWIW: https://readthedocs.org/projects/dagger-io/builds/ - Click on Publish release
- Tag & publish:
cd sdk/typescript && export TYPESCRIPT_SDK_VERSION=$(changie latest) && cd ../..
git tag "sdk/typescript/${TYPESCRIPT_SDK_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push "${DAGGER_REPO_REMOTE:?must be set}" sdk/typescript/${TYPESCRIPT_SDK_VERSION}This will trigger the Publish TypeScript SDK
workflow
which publishes a new version to β¬’ npmjs.com/package/@dagger.io/dagger
- Upload the release notes by running:
gh release create "sdk/typescript/${TYPESCRIPT_SDK_VERSION:?must be set}" \
--draft --verify-tag --title sdk/typescript/$TYPESCRIPT_SDK_VERSION \
--notes-file sdk/typescript/.changes/$TYPESCRIPT_SDK_VERSION.md- Check that release notes look good in
Preview -
β οΈ De-select Set as the latest release (only used for π Engine + π CLI releases) - Click on Publish release
- Tag & publish:
cd sdk/elixir && export ELIXIR_SDK_VERSION=$(changie latest) && cd ../..
git tag "sdk/elixir/${ELIXIR_SDK_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push "${DAGGER_REPO_REMOTE:?must be set}" sdk/elixir/${ELIXIR_SDK_VERSION}This will trigger the Publish Elixir SDK
workflow
which publishes a new version to π§ͺ hex.pm/packages/dagger
- Upload the release notes by running:
gh release create "sdk/elixir/${ELIXIR_SDK_VERSION:?must be set}" \
--draft --verify-tag --title sdk/elixir/$ELIXIR_SDK_VERSION \
--notes-file sdk/elixir/.changes/$ELIXIR_SDK_VERSION.md- Check that release notes look good in
Preview -
β οΈ De-select Set as the latest release (only used for π Engine + π CLI releases) - Click on Publish release
- Tag & publish:
cd sdk/php && export PHP_SDK_VERSION=$(changie latest) && cd ../..
git tag "sdk/php/${PHP_SDK_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push "${DAGGER_REPO_REMOTE:?must be set}" sdk/php/${PHP_SDK_VERSION}This will trigger the Publish PHP SDK
workflow
which publishes to
github.com/dagger/dagger-php-sdk.
- Upload the release notes by running:
gh release create "sdk/php/${PHP_SDK_VERSION:?must be set}" \
--draft --verify-tag --title sdk/php/$PHP_SDK_VERSION \
--notes-file sdk/php/.changes/$PHP_SDK_VERSION.md- Check that release notes look good in
Preview -
β οΈ De-select Set as the latest release (only used for π Engine + π CLI releases) - Click on Publish release
- Tag & publish:
export HELM_CHART_VERSION="$(awk '/^version: / { print $2 }' helm/dagger/Chart.yaml)"
git tag "helm/chart/v${HELM_CHART_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push "${DAGGER_REPO_REMOTE:?must be set}" "helm/chart/v${HELM_CHART_VERSION:?must be set}"This will trigger the publish-helm-chart
workflow
which publishes to π
registry.dagger.io/dagger-helm.
After the Engine+SDKs are all released from the release-vX.Y.Z branch, you will need to "forward-port" a few of the commits there up to main. Namely:
- The commits from the release notes PR, e.g. when patching
v0.11.8tov0.11.9, this commit - The commits from the SDK engine version bump PR, e.g. when patching
v0.11.8tov0.11.9, the commits from this PR - The commits from the CI engine version bump PR, e.g. when patching
v0.11.8tov0.11.9, this commit
One easy way to do this is to re-use the engine version bump PR that was created against main earlier, cherry-picking in the missing commits.
Be sure to use "Rebase and Merge" when merging the PR to main to preserve the history of the cherry-picked commits.
Warning
Merging a documentation PR does NOT automatically deploy the new documentation to the production website.
There are two websites for documentation:
- Staging: https://devel.docs.dagger.io - Netlify dashboard
- Production: https://docs.dagger.io - Netlify dashboard
When a PR is merged, a new deployment is created for the documentation site and it is automatically published to https://devel.docs.dagger.io via Netlify.
Use this staging website to test the documentation, including:
- verifying that the new content appears in the navigation
- verifying internal and external links work correctly
- verifying that images appear correctly
- etc.
When a PR is merged, a new production deployment is also created for https://docs.dagger.io. However, this deployment is not automatically published.
After testing the documentation using the staging website and if you are satisfied with it, manually publish the production deployment via Netlify as follows:
- Log in to the Netlify dashboard for https://docs.dagger.io.
- Refer to the list of "production deploys" and select the one you wish to deploy. Usually, this will be the most recent one. You can confirm this by checking the deployment hash against the latest commit hash in the dagger/dagger repository main branch.
- On the deployment page, click the "Preview" button to once again preview/check the deployment. You can also check the deployment log to confirm there were no errors during the documentation build process.
- If you are satisfied with the preview, click the "Publish deploy" button. This will publish the selected deployment on https://docs.dagger.io
Note
There have been cases where Netlify builds have failed with errors, but the same build succeeds when performed locally. In the past, one reason for this has been Netlify's use of a stale cache. In case you encounter this error, click "Options -> Clear cache and retry with latest branch commit" to recreate the deployment with a clean cache.
The Dagger Playground is set to automatically update once there's a new release of the Dagger Engine.
- Mention in the release thread on Discord that Playground can be updated to the just-released version. cc @marcosnils @matipan @gerhard
- Mention in the release thread on Discord that Playground can be updated to the just-released version. cc @marcosnils @matipan @grouville
- Mention in the release thread on Discord that Dagger Cloud can be updated to the just-released version. cc @marcosnils @matipan @sipsma
- If the install scripts
install.shorinstall.ps1have changed since the last release, they must be manually updated on Amazon S3 (CloudFront should also be manually invalidated). cc @gerhard
- Submit PR with the version bump, e.g. dagger/dagger-for-github#123
- Ask @gerhard or @jpadams to review it
Tip
We should automate the above mentioned steps same as we do with the PR which bumps the Engine version, e.g. dagger#7318
- Force update the major version, currently
v5
git tag --sign --force v5
# Update the date in the comment to e.g. 2024-05-08
git push origin v5 --force- Once this PR is merged, tag the new version
# Find the latest released patch https://github.com/dagger/dagger-for-github/releases
git tag --sign <NEXT_PATCH_VERSION>
# Use the date as the comment, e.g. 2024-05-08
git push origin <NEXT_PATCH_VERSION> --force- Create a new release from the patch tag (auto-fill release notes via the GitHub UI)
- Check that Dagger Homebrew formula has been updated to latest, e.g. dagger 0.10.2. This is automated, but note that it may take several hours to trigger.
- Check that Dagger nix flake has been updated to latest, e.g. dagger: -> v0.10.2
- When all the above done, remember to add the
RELEASING.mdchanges to theimprove-releasing-during-v...PR that you have opened earlier (remember to toggle all the checkboxes back to[ ]). Here is an example: dagger#5658 - Close the Discord release thread you created in Let the team know
- Close the GitHub milestone you created in Let the team know
- If there are remaining PRs/issues that were not resolved, then move them into the next milestone (or remove it from a milestone entirely)