Skip to content

Commit

Permalink
Update uv, pip and pre-commit versions automatically in more places. (#…
Browse files Browse the repository at this point in the history 
…45398)
  • Loading branch information
@potiuk
potiuk authored Jan 4, 2025
1 parent 345fd84 commit de6d83a
Show file tree
Hide file tree
Showing 10 changed files with 260 additions and 201 deletions.
7 changes: 3 additions & 4 deletions .github/actions/install-pre-commit/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,18 @@
name: 'Install pre-commit'
description: 'Installs pre-commit and related packages'
inputs:
# TODO(potiuk): automate update of these versions
python-version:
description: 'Python version to use'
default: "3.9"
uv-version:
description: 'uv version to use'
default: "0.5.14"
default: "0.5.14" # Keep this comment to allow automatic replacement of uv version
pre-commit-version:
description: 'pre-commit version to use'
default: "4.0.1"
default: "4.0.1" # Keep this comment to allow automatic replacement of pre-commit version
pre-commit-uv-version:
description: 'pre-commit-uv version to use'
default: "4.1.4"
default: "4.1.4" # Keep this comment to allow automatic replacement of pre-commit-uv version
runs:
using: "composite"
steps:
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/basic-tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -307,11 +307,12 @@ jobs:
run: >
pre-commit run
--all-files --show-diff-on-failure --color always --verbose
--hook-stage manual update-installers || true
--hook-stage manual update-installers-and-pre-commit || true
if: always()
env:
UPGRADE_UV: "true"
UPGRADE_PIP: "false"
UPGRADE_PRE_COMMIT: "true"
- name: "Run automated upgrade for pip"
run: >
pre-commit run
Expand Down
8 changes: 4 additions & 4 deletions .pre-commit-config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -197,12 +197,12 @@ repos:
additional_dependencies: ['pyyaml']
pass_filenames: false
require_serial: true
- id: update-installers
name: Update installers to latest (manual)
entry: ./scripts/ci/pre_commit/update_installers.py
- id: update-installers-and-pre-commit
name: Update installers and pre-commit to latest (manual)
entry: ./scripts/ci/pre_commit/update_installers_and_pre_commit.py
stages: ['manual']
language: python
files: ^.pre-commit-config.yaml$|^scripts/ci/pre_commit/update_installers.py$
files: ^.pre-commit-config.yaml$|^scripts/ci/pre_commit/update_installers_and_pre_commit.py$
pass_filenames: false
require_serial: true
additional_dependencies: ['pyyaml', 'rich>=12.4.4', 'requests']
Expand Down
2 changes: 1 addition & 1 deletion contributing-docs/08_static_code_checks.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -374,7 +374,7 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------+---------+
| update-installed-providers-to-be-sorted | Sort and uniquify installed_providers.txt | |
+-----------------------------------------------------------+--------------------------------------------------------+---------+
| update-installers | Update installers to latest (manual) | |
| update-installers-and-pre-commit | Update installers and pre-commit to latest (manual) | |
+-----------------------------------------------------------+--------------------------------------------------------+---------+
| update-local-yml-file | Update mounts in the local yml file | |
+-----------------------------------------------------------+--------------------------------------------------------+---------+
Expand Down
56 changes: 29 additions & 27 deletions dev/breeze/doc/ci/02_images.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -419,33 +419,35 @@ DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
The following build arguments (`--build-arg` in docker build command)
can be used for CI images:

| Build argument | Default value | Description |
|-----------------------------------|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `PYTHON_BASE_IMAGE` | `python:3.9-slim-bookworm` | Base Python image |
| `PYTHON_MAJOR_MINOR_VERSION` | `3.9` | major/minor version of Python (should match base image) |
| `DEPENDENCIES_EPOCH_NUMBER` | `2` | increasing this number will reinstall all apt dependencies |
| `ADDITIONAL_PIP_INSTALL_FLAGS` | | additional `pip` flags passed to the installation commands (except when reinstalling `pip` itself) |
| `HOME` | `/root` | Home directory of the root user (CI image has root user as default) |
| `AIRFLOW_HOME` | `/root/airflow` | Airflow's HOME (that's where logs and sqlite databases are stored) |
| `AIRFLOW_SOURCES` | `/opt/airflow` | Mounted sources of Airflow |
| `AIRFLOW_REPO` | `apache/airflow` | the repository from which PIP dependencies are pre-installed |
| `AIRFLOW_BRANCH` | `main` | the branch from which PIP dependencies are pre-installed |
| `AIRFLOW_CI_BUILD_EPOCH` | `1` | increasing this value will reinstall PIP dependencies from the repository from scratch |
| `AIRFLOW_CONSTRAINTS_LOCATION` | | If not empty, it will override the source of the constraints with the specified URL or file. |
| `AIRFLOW_CONSTRAINTS_REFERENCE` | | reference (branch or tag) from GitHub repository from which constraints are used. By default it is set to `constraints-main` but can be `constraints-2-X`. |
| `AIRFLOW_EXTRAS` | `all` | extras to install |
| `UPGRADE_INVALIDATION_STRING` | | If set to any random value the dependencies are upgraded to newer versions. In CI it is set to build id. |
| `ADDITIONAL_AIRFLOW_EXTRAS` | | additional extras to install |
| `ADDITIONAL_PYTHON_DEPS` | | additional Python dependencies to install |
| `DEV_APT_COMMAND` | | Dev apt command executed before dev deps are installed in the first part of image |
| `ADDITIONAL_DEV_APT_COMMAND` | | Additional Dev apt command executed before dev dep are installed in the first part of the image |
| `DEV_APT_DEPS` | | Dev APT dependencies installed in the first part of the image (default empty means default dependencies are used) |
| `ADDITIONAL_DEV_APT_DEPS` | | Additional apt dev dependencies installed in the first part of the image |
| `ADDITIONAL_DEV_APT_ENV` | | Additional env variables defined when installing dev deps |
| `AIRFLOW_PIP_VERSION` | `24.3.1` | PIP version used. |
| `AIRFLOW_UV_VERSION` | `0.5.14` | UV version used. |
| `AIRFLOW_USE_UV` | `true` | Whether to use UV for installation. |
| `PIP_PROGRESS_BAR` | `on` | Progress bar for PIP installation |
| Build argument | Default value | Description |
|---------------------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------|
| `PYTHON_BASE_IMAGE` | `python:3.9-slim-bookworm` | Base Python image |
| `PYTHON_MAJOR_MINOR_VERSION` | `3.9` | major/minor version of Python (should match base image) |
| `DEPENDENCIES_EPOCH_NUMBER` | `2` | increasing this number will reinstall all apt dependencies |
| `ADDITIONAL_PIP_INSTALL_FLAGS` | | additional `pip` flags passed to the installation commands (except when reinstalling `pip` itself) |
| `HOME` | `/root` | Home directory of the root user (CI image has root user as default) |
| `AIRFLOW_HOME` | `/root/airflow` | Airflow's HOME (that's where logs and sqlite databases are stored) |
| `AIRFLOW_SOURCES` | `/opt/airflow` | Mounted sources of Airflow |
| `AIRFLOW_REPO` | `apache/airflow` | the repository from which PIP dependencies are pre-installed |
| `AIRFLOW_BRANCH` | `main` | the branch from which PIP dependencies are pre-installed |
| `AIRFLOW_CI_BUILD_EPOCH` | `1` | increasing this value will reinstall PIP dependencies from the repository from scratch |
| `AIRFLOW_CONSTRAINTS_LOCATION` | | If not empty, it will override the source of the constraints with the specified URL or file. |
| `AIRFLOW_CONSTRAINTS_REFERENCE` | `constraints-main` | reference (branch or tag) from GitHub repository from which constraints are used. |
| `AIRFLOW_EXTRAS` | `all` | extras to install |
| `UPGRADE_INVALIDATION_STRING` | | If set to any random value the dependencies are upgraded to newer versions. In CI it is set to build id. |
| `ADDITIONAL_AIRFLOW_EXTRAS` | | additional extras to install |
| `ADDITIONAL_PYTHON_DEPS` | | additional Python dependencies to install |
| `DEV_APT_COMMAND` | | Dev apt command executed before dev deps are installed in the first part of image |
| `ADDITIONAL_DEV_APT_COMMAND` | | Additional Dev apt command executed before dev dep are installed in the first part of the image |
| `DEV_APT_DEPS` | | Dev APT dependencies installed in the first part of the image (default empty means default dependencies are used) |
| `ADDITIONAL_DEV_APT_DEPS` | | Additional apt dev dependencies installed in the first part of the image |
| `ADDITIONAL_DEV_APT_ENV` | | Additional env variables defined when installing dev deps |
| `AIRFLOW_PIP_VERSION` | `24.3.1` | `pip` version used. |
| `AIRFLOW_UV_VERSION` | `0.5.14` | `uv` version used. |
| `AIRFLOW_PRE_COMMIT_VERSION` | `4.0.1` | `pre-commit` version used. |
| `AIRFLOW_PRE_COMMIT_UV_VERSION` | `4.1.4` | `pre-commit-uv` version used. |
| `AIRFLOW_USE_UV` | `true` | Whether to use UV for installation. |
| `PIP_PROGRESS_BAR` | `on` | Progress bar for PIP installation |


Here are some examples of how CI images can built manually. CI is always
Expand Down
Loading

0 comments on commit de6d83a

Please sign in to comment.