Skip to content

Commit

Permalink
Update piplock renewal gh action
Browse files Browse the repository at this point in the history
Remove old obsolete files

Add git-ignore file to avoid upload Pipfile.lock by the user since they will be created only via gha

Include codeserver notebook on the refresh-piplock-files recipe

Add git-ignore to the intel flavor and modify the makefile refresh-piplock to work with optional directories

Update refresh-pipfilelock-files to accept optional file directories for updates

Update documentation in regards the Pipfile lock generation
  • Loading branch information
atheo89 committed Dec 11, 2024
1 parent 227bfb7 commit 293f74c
Show file tree
Hide file tree
Showing 52 changed files with 400 additions and 222 deletions.
47 changes: 0 additions & 47 deletions .github/workflows/piplock-renewal-2023a.yml

This file was deleted.

47 changes: 0 additions & 47 deletions .github/workflows/piplock-renewal-2023b.yml

This file was deleted.

47 changes: 0 additions & 47 deletions .github/workflows/piplock-renewal-2024a.yml

This file was deleted.

47 changes: 0 additions & 47 deletions .github/workflows/piplock-renewal-main.yml

This file was deleted.

65 changes: 65 additions & 0 deletions .github/workflows/piplock-renewal.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
# This GitHub action is meant to update the pipfile.locks
name: Pipfile.locks Renewal Action

on: # yamllint disable-line rule:truthy
workflow_dispatch: # for manual trigger workflow from GH Web UI
inputs:
branch:
description: 'Specify branch'
required: false
default: 'main'
python_version:
description: 'Select Python version to update Pipfile.lock'
required: false
default: '3.11'
type: choice
options:
- '3.11'
- '3.9'
- '3.8'
update_optional_dirs:
description: 'Include optional directories in update'
required: false
default: 'false'
type: choice
options:
- 'true'
- 'false'

jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
# Checkout the specified branch from the specified organization
- name: Checkout code from the specified branch
uses: actions/checkout@v4
with:
ref: ${{ github.event.inputs.branch }}
token: ${{ secrets.GH_ACCESS_TOKEN }}

# Configure Git
- name: Configure Git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git config --global user.name "GitHub Actions"
# Setup Python environment with the specified version (or default to '3.11')
- name: Setup Python environment
uses: actions/setup-python@v5
with:
python-version: ${{ github.event.inputs.python_version }}

# Install pipenv
- name: Install pipenv
run: pip install pipenv

# Run makefile recipe to refresh Pipfile.lock and push changes back to the branch
- name: Run make refresh-pipfilelock-files and push the changes back to the branch
run: |
make refresh-pipfilelock-files PYTHON_VERSION=${{ github.event.inputs.python_version }} INCLUDE_OPT_DIRS=${{ github.event.inputs.update_optional_dirs }}
git add -f .
git commit -m "Update Pipfile.lock files by piplock-renewal.yaml action"
git push origin ${{ github.event.inputs.branch }}
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ Pull requests are the best way to propose changes to the notebooks repository:
- Create a proper filepath and naming to the corresponding folder
- Add the minimum files you have to add:
- Pipfile with the additional packages
- Generate the Pipfile lock by running `pipenv lock` to the corresponding pipfile directory
- Dockefile with proper instructions
- Kustomization objects to deploy the new notebook into an openshift cluster (Kustomization.yaml, service.yaml, statefulset.yaml)
- Create instructions into Makefile, for example if you derive the new notebooks from minimal then the recipe should be like the following:
Expand All @@ -38,6 +37,7 @@ Pull requests are the best way to propose changes to the notebooks repository:
$(call image,$@,jupyter/${NOTEBOOK_NAME}/ubi8-python-3.8,$<)
```
- Add the paths of the new pipfiles under `refresh-pipfilelock-files`
- Run the [piplock-renewal.yaml](https://github.com/opendatahub-io/notebooks/blob/main/.github/workflows/piplock-renewal.yaml) against your fork branch, check [here](https://github.com/opendatahub-io/notebooks/blob/main/README.md) for more info.
- Test the changes locally, by manually running the `$ make jupyter-${NOTEBOOK_NAME}-ubi8-python-3.8` from the terminal.
Expand Down
88 changes: 55 additions & 33 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -636,41 +636,63 @@ validate-rstudio-image: bin/kubectl
continue; \
fi; \

# This is only for the workflow action
# This recipe used mainly from the Pipfile.locks Renewal Action
# Default Python version
PYTHON_VERSION ?= 3.11
ROOT_DIR := $(shell pwd)
BASE_DIRS := base/c9s-python-$(PYTHON_VERSION) \
base/ubi9-python-$(PYTHON_VERSION) \
jupyter/minimal/ubi9-python-$(PYTHON_VERSION) \
jupyter/datascience/ubi9-python-$(PYTHON_VERSION) \
jupyter/pytorch/ubi9-python-$(PYTHON_VERSION) \
jupyter/tensorflow/ubi9-python-$(PYTHON_VERSION) \
jupyter/trustyai/ubi9-python-$(PYTHON_VERSION) \
jupyter/rocm/tensorflow/ubi9-python-$(PYTHON_VERSION) \
jupyter/rocm/pytorch/ubi9-python-$(PYTHON_VERSION) \
codeserver/ubi9-python-$(PYTHON_VERSION) \
runtimes/minimal/ubi9-python-$(PYTHON_VERSION) \
runtimes/datascience/ubi9-python-$(PYTHON_VERSION) \
runtimes/pytorch/ubi9-python-$(PYTHON_VERSION) \
runtimes/tensorflow/ubi9-python-$(PYTHON_VERSION) \
runtimes/rocm-tensorflow/ubi9-python-$(PYTHON_VERSION) \
runtimes/rocm-pytorch/ubi9-python-$(PYTHON_VERSION)

# Default value is false, can be overiden
# The below directories are not supported on tier-1
INCLUDE_OPT_DIRS ?=false
OPT-DIRS := jupyter/intel/ml/ubi9-python-$(PYTHON_VERSION) \
jupyter/intel/pytorch/ubi9-python-$(PYTHON_VERSION) \
jupyter/intel/tensorflow/ubi9-python-$(PYTHON_VERSION) \
intel/runtimes/ml/ubi9-python-$(PYTHON_VERSION) \
intel/runtimes/pytorch/ubi9-python-$(PYTHON_VERSION) \
intel/runtimes/tensorflow/ubi9-python-$(PYTHON_VERSION)

# This recipe gets args, can be used like
# make refresh-pipfilelock-files PYTHON_VERSION=3.11 INCLUDE_OPT_DIRS=false
.PHONY: refresh-pipfilelock-files
refresh-pipfilelock-files:
cd base/ubi9-python-3.9 && pipenv lock
cd base/c9s-python-3.9 && pipenv lock
cd jupyter/minimal/ubi9-python-3.9 && pipenv lock
cd jupyter/datascience/ubi9-python-3.9 && pipenv lock
cd jupyter/pytorch/ubi9-python-3.9 && pipenv lock
cd jupyter/tensorflow/ubi9-python-3.9 && pipenv lock
cd jupyter/trustyai/ubi9-python-3.9 && pipenv lock
cd runtimes/minimal/ubi9-python-3.9 && pipenv lock
cd runtimes/datascience/ubi9-python-3.9 && pipenv lock
cd runtimes/pytorch/ubi9-python-3.9 && pipenv lock
cd runtimes/tensorflow/ubi9-python-3.9 && pipenv lock
cd runtimes/rocm-tensorflow/ubi9-python-3.9 && pipenv lock
cd runtimes/rocm-pytorch/ubi9-python-3.9 && pipenv lock
cd base/c9s-python-3.11 && pipenv lock
cd base/ubi9-python-3.11 && pipenv lock
cd codeserver/ubi9-python-3.11 && pipenv lock
cd jupyter/minimal/ubi9-python-3.11 && pipenv lock
cd jupyter/datascience/ubi9-python-3.11 && pipenv lock
cd jupyter/pytorch/ubi9-python-3.11 && pipenv lock
cd jupyter/tensorflow/ubi9-python-3.11 && pipenv lock
cd jupyter/trustyai/ubi9-python-3.11 && pipenv lock
cd jupyter/rocm/tensorflow/ubi9-python-3.11 && pipenv lock
cd jupyter/rocm/pytorch/ubi9-python-3.11 && pipenv lock
cd runtimes/minimal/ubi9-python-3.11 && pipenv lock
cd runtimes/datascience/ubi9-python-3.11 && pipenv lock
cd runtimes/pytorch/ubi9-python-3.11 && pipenv lock
cd runtimes/tensorflow/ubi9-python-3.11 && pipenv lock
cd runtimes/rocm-tensorflow/ubi9-python-3.11 && pipenv lock
cd runtimes/rocm-pytorch/ubi9-python-3.11 && pipenv lock



@echo "Updating Pipfile.lock files for Python $(PYTHON_VERSION)"
@if [ "$(INCLUDE_OPT_DIRS)" = "true" ]; then \
echo "Including optional directories"; \
DIRS="$(BASE_DIRS) $(OPT_DIRS)"; \
else \
DIRS="$(BASE_DIRS)"; \
fi; \
for dir in $$DIRS; do \
echo "Processing directory: $$dir"; \
cd $(ROOT_DIR); \
if [ -d "$$dir" ]; then \
echo "Updating $(PYTHON_VERSION) Pipfile.lock in $$dir"; \
cd $$dir; \
if [ -f "Pipfile" ]; then \
pipenv lock; \
else \
echo "No Pipfile found in $$dir, skipping."; \
fi; \
else \
echo "Skipping $$dir as it does not exist"; \
fi; \
done

# This is only for the workflow action
# For running manually, set the required environment variables
Expand Down
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,13 @@ Use podman/docker to execute the workbench images as container.
podman run -it -p 8888:8888 quay.io/opendatahub/workbench-images:jupyter-minimal-ubi9-python-3.9-2024a-20240317-6f4c36b
```

### Pipfile.lock Generation

To maintain consistency across different local environments and prevent platform-specific issues, users should not push their Pipfile.lock files after local execution. Collaborators may encounter discrepancies due to varying local setups, leading to potential conflicts during the lock file generation. Instead, users can update these files through the [piplock-renewal.yaml](https://github.com/opendatahub-io/notebooks/blob/main/.github/workflows/piplock-renewal.yaml) GitHub Action.

This action allows users to specify a branch for updating and automerging the Pipfile.lock files, select the desired Python version, and choose whether to include optional directories in the update.
*NOTE:* To ensure the GitHub Action runs successfully, users must create a `GH_ACCESS_TOKEN` secret in their fork.

### Deploy & Test

#### Running Python selftests in Pytest
Expand Down
6 changes: 6 additions & 0 deletions base/c9s-python-3.11/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
Pipfile.lock
Loading

0 comments on commit 293f74c

Please sign in to comment.