Skip to content

Commit 149d760

Browse files
madhavajayreplomancer
madhavajay
and
replomancer
authored
Fixing CI (OpenMined#363)
* Touch * Touch back * Updated GitHub CI - Ran black black-nb mypy - Removed python 3.5 - Fixed issue with MacOS Build using --features=-supports_dynamic_linker - Updated Google DP to the latest commit - Renamed Bazel target from bindings_test to pydp - Updated pyproject.toml - Fixed com_google_protobuf issue in WORKSPACE - Disabled docker build on every commit / push * Added pre-commit yaml file - Ran pre-commit to fix lots of code issues - Removed unused # type: ignore - Updated setup.cfg with isort, mypy and flake8 rules - Ran isort - Disabled notebook tests - Fixed matrix.os template variable in tests.yml * Update docker to use python 3.9 - Fix mypy issues with missing typeshed - Fixing tests.yml * Fixing Dockerfile path to bazelisk - Fixing issue with converting matrix.os to bazel config name * Add ubuntu to .bazelrc file * Fixing matrix.os conversion to bazel config * Fixing multi-os issues in tests.yml * More multi-os fixes * Fixing windows bazel * Fix windows * Adding bazel caching for linux - Fixing install wheel on windows - Fixing print version * More CI fixes * tunshell * Fixing macos wheel * remove tunshell * Quickfix serialized test data * Added bazel-cache - Combined publish actions into a single github action - Added nightly tests for older versions of python - Updated some package meta-data - Bumped version to 1.0.3 - Added poetry.lock file to speed up CI * Removed unused exclusion rule from pre-commit * Adding other OSes to the Bazel Cache step * Update test to include information about cached result from tests/algorithms/test_bounded_mean/test_bounded_mean_int64_data.proto * Refactored Docker and Docs actions - Updated poetry lock file * Renaming a few action names - Testing versions cron * tunshell into windows machine * tunshell into mac * Revert cron job for versions - Remove tunshell for mac * Trying specific version of cache plugin for windows tar cache error * Added proper bazel cache directories for each OS plus Windows fix * Trying different tar.exe fix for Windows * Forgot to remove --output_base ./bazel-cache * Trying windows tar fix - Busting cache on linux and mac * Trying older bazel cache strategy * Debugging Windows Bazel Cache * Trying actions/[email protected] * Trying a different windows bazel cache approach * Fix test yaml - Trying Windows Bash tunshell * woops checkout not cache * Wrong runner * Windows good times * Windows cache might be working * Try without symlinks? * Putting cache code in all workflows - Time to test Windows cache retrieval * Added linting requirement back to main tests * Fixed PR change requests * Replace bazel with bazelisk - Use dp submodule SHA * debugging versions check * Adding python binary and site-packages for different versions on unix * Test all versions on mac and linux * Adding Windows support for python path detection * Reversed quotes and removed \ line endings * Changing the var name on windows * Using old syntax for the PYTHONPATH on windows * Re-enabled windows on versions * Replacing incorrect Capital R in replace Co-authored-by: Lukasz Kozuchowski <[email protected]>
1 parent 024c27e commit 149d760

File tree

83 files changed

+8427
-3864
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

83 files changed

+8427
-3864
lines changed

‎.bazelrc

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
build:Linux --cxxopt='-std=c++17'
2-
build:macOS --cxxopt='-std=c++17'
3-
build:Windows --cxxopt='/std:c++17'
1+
build:linux --cxxopt='-std=c++17'
2+
build:macos --cxxopt='-std=c++17' --features=-supports_dynamic_linker
3+
build:windows --cxxopt='/std:c++17'

‎.dockerignore

+2
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,4 @@
11
.git
22
/third_party
3+
/build
4+
/bazel-*

‎.github/workflows/build-docs.yml

+112
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
name: Build Dev Docs
2+
3+
on:
4+
push:
5+
branches:
6+
- dev
7+
8+
jobs:
9+
build-docs:
10+
strategy:
11+
fail-fast: false
12+
max-parallel: 1
13+
matrix:
14+
os: [ubuntu-latest]
15+
python-version: [3.9]
16+
runs-on: ${{ matrix.os }}
17+
steps:
18+
- uses: actions/checkout@v2
19+
with:
20+
submodules: true
21+
22+
- name: Set up Python ${{ matrix.python-version }}
23+
uses: actions/setup-python@v2
24+
with:
25+
python-version: ${{ matrix.python-version }}
26+
27+
- name: Cache Bazel Unix
28+
uses: actions/[email protected]
29+
with:
30+
path: |
31+
./bazel-PyDP
32+
./bazel-bin
33+
./bazel-cache
34+
./bazel-out
35+
key: ${{ runner.os }}-bazel-${{ hashFiles('WORKSPACE') }}
36+
37+
- name: Build Google DP Unix
38+
if: runner.os != 'Windows'
39+
timeout-minutes: 20
40+
run: |
41+
PYTHONHOME=$(which python)
42+
PYTHONPATH=$(python -c "import sys; print([x for x in sys.path if 'site-packages' in x][0]);")
43+
BAZEL_CONFIG_OS=$(python -c "print('${{ matrix.os }}'.split('-')[0].lower().replace('ubuntu', 'linux'))")
44+
echo "Running: ${{ matrix.os }}"
45+
echo "Using BAZEL_CONFIG_OS: $BAZEL_CONFIG_OS"
46+
bazel --output_base ./bazel-cache build src/python:pydp \
47+
--config $BAZEL_CONFIG_OS \
48+
--verbose_failures --action_env=PYTHON_BIN_PATH=$PYTHONHOME \
49+
--action_env=PYTHON_LIB_PATH=$PYTHONPATH
50+
cp -f ./bazel-bin/src/bindings/_pydp.so ./src/pydp
51+
52+
- name: Upgrade pip
53+
run: |
54+
pip install --upgrade --user pip
55+
56+
- name: Install Poetry
57+
run: |
58+
pip install poetry
59+
60+
- name: Get poetry cache dir
61+
id: poetry-cache
62+
run: |
63+
echo "::set-output name=dir::$(poetry config cache-dir)"
64+
65+
- name: poetry cache
66+
uses: actions/cache@v2
67+
with:
68+
path: ${{ steps.poetry-cache.outputs.dir }}
69+
key: ${{ runner.os }}-pip-py${{ matrix.python-version }}-${{ hashFiles('**/pyproject.toml') }}
70+
restore-keys: |
71+
${{ runner.os }}-pip-py${{ matrix.python-version }}-
72+
73+
- name: Install dependencies
74+
run: |
75+
poetry install
76+
77+
- name: Build PyDP Linux / Windows
78+
if: runner.os != 'macOS'
79+
run: |
80+
poetry run python setup.py build bdist_wheel
81+
82+
- name: Install Wheel Unix
83+
if: runner.os != 'Windows'
84+
run: |
85+
pip install `find -L ./ -name "*.whl"`
86+
87+
- name: Import Package
88+
run: |
89+
python -c "import pydp; print(pydp.__version__)"
90+
91+
- uses: ammaraskar/sphinx-action@master
92+
with:
93+
docs-folder: "docs/"
94+
95+
- name: Commit documentation changes
96+
run: |
97+
git clone https://github.com/ammaraskar/sphinx-action-test.git --branch gh-pages --single-branch gh-pages
98+
cp -r docs/_build/html/* gh-pages/
99+
cd gh-pages
100+
git config --local user.email "[email protected]"
101+
git config --local user.name "GitHub Action"
102+
git add .
103+
git commit -m "Update documentation" -a || true
104+
# The above command will fail if no changes were present, so we ignore that
105+
106+
- name: Push changes
107+
uses: ad-m/github-push-action@master
108+
with:
109+
branch: gh-pages
110+
directory: gh-pages
111+
force: true
112+
github_token: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/build_dev_docs.yml

-57
This file was deleted.

‎.github/workflows/publish-docker.yml

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: Publish Docker Image
2+
on:
3+
release:
4+
types: [published]
5+
6+
jobs:
7+
publish:
8+
name: Build Docker Container
9+
runs-on: ubuntu-latest
10+
strategy:
11+
matrix:
12+
python-version: [3.6, 3.7, 3.8, 3.9]
13+
env:
14+
IMAGE: openmined/pydp
15+
16+
steps:
17+
- uses: actions/checkout@v2
18+
with:
19+
submodules: true
20+
- name: Set up Python ${{ matrix.python-version }}
21+
uses: actions/setup-python@v2
22+
with:
23+
python-version: ${{ matrix.python-version }}
24+
25+
- name: Show python version and build image name
26+
id: build_name
27+
run: |
28+
python -c "import sys; print(sys.version)"
29+
echo "::set-output name=image_name::${IMAGE}-${GITHUB_REF##*/}-${{matrix.python-version}}"
30+
31+
- name: Build images
32+
run: docker build -t ${{steps.build_name.outputs.image_name}} --build-arg PYTHON_VERSION=${{matrix.python-version}} -f Dockerfile .
33+
34+
- name: Log into Docker
35+
run: echo "${{ secrets.DOCKER_PASSWORD }}" | docker login -u ${{ secrets.DOCKER_LOGIN }} --password-stdin
36+
37+
- name: Tag and Push Image
38+
run: |
39+
docker tag ${{steps.build_name.outputs.image_name}} $IMAGE:${{ github.event.release.tag_name }}-py${{ matrix.python-version }}
40+
docker push $IMAGE:${{ github.event.release.tag_name }}-py${{ matrix.python-version }}
41+
42+
- name: Tag Python 3.9 as latest
43+
if: matrix.python-version == '3.9'
44+
run: |
45+
docker tag ${{steps.build_name.outputs.image_name}} $IMAGE:${{ github.event.release.tag_name }}
46+
docker push $IMAGE:${{ github.event.release.tag_name }}
47+
docker tag ${{steps.build_name.outputs.image_name}} $IMAGE:latest
48+
docker push $IMAGE:latest

‎.github/workflows/publish.yml

+165
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
name: Publish PyDP
2+
3+
on:
4+
release:
5+
types: [published]
6+
7+
jobs:
8+
deploy:
9+
strategy:
10+
fail-fast: false
11+
max-parallel: 12
12+
matrix:
13+
os: [ubuntu-latest, macos-latest, windows-latest]
14+
python-version: [3.6, 3.7, 3.8, 3.9]
15+
runs-on: ${{ matrix.os }}
16+
17+
steps:
18+
- uses: actions/checkout@v2
19+
with:
20+
submodules: true
21+
22+
- name: Set up Python ${{ matrix.python-version }}
23+
uses: actions/setup-python@v2
24+
with:
25+
python-version: ${{ matrix.python-version }}
26+
27+
- name: Setup msbuild Windows
28+
if: runner.os == 'Windows'
29+
uses: microsoft/[email protected]
30+
31+
- name: Fix Paths Windows
32+
# Make sure that tar.exe from Git is used not from Windows
33+
if: runner.os == 'Windows'
34+
run: |
35+
@("C:\Program Files\Git\usr\bin") + (Get-Content $env:GITHUB_PATH) | Set-Content $env:GITHUB_PATH -Encoding utf8
36+
37+
- name: Cache Bazel Unix
38+
# Not working on Windows: https://github.com/actions/cache/issues/576
39+
if: runner.os != 'Windows'
40+
uses: actions/[email protected]
41+
with:
42+
path: |
43+
./bazel-PyDP
44+
./bazel-bin
45+
./bazel-cache
46+
./bazel-out
47+
key: ${{ runner.os }}-bazel-${{ hashFiles('WORKSPACE') }}
48+
49+
- name: Update environment variables Windows
50+
if: runner.os == 'Windows'
51+
# See:
52+
# - On Bazel cache: https://docs.bazel.build/versions/master/output_directories.html
53+
run: |
54+
echo "BAZEL_CACHE_DIR=$env:USERPROFILE\_bazel_$env:USERNAME" >> $env:GITHUB_ENV
55+
56+
- name: Cache Bazel Windows
57+
if: runner.os == 'Windows'
58+
# https://stackoverflow.com/questions/66870002/github-actions-cache-maven-m2-repository-on-windows-environment-c-users-run
59+
uses: actions/[email protected]
60+
with:
61+
path: |
62+
./bazel-cache
63+
key: ${{ runner.os }}-bazel-${{ hashFiles('WORKSPACE') }}
64+
65+
- name: Build Google DP Unix
66+
if: runner.os != 'Windows'
67+
timeout-minutes: 20
68+
run: |
69+
PYTHONHOME=$(which python)
70+
PYTHONPATH=$(python -c "import sys; print([x for x in sys.path if 'site-packages' in x][0]);")
71+
BAZEL_CONFIG_OS=$(python -c "print('${{ matrix.os }}'.split('-')[0].lower().replace('ubuntu', 'linux'))")
72+
echo "Running: ${{ matrix.os }}"
73+
echo "Using BAZEL_CONFIG_OS: $BAZEL_CONFIG_OS"
74+
bazel --output_base ./bazel-cache build src/python:pydp \
75+
--config $BAZEL_CONFIG_OS \
76+
--verbose_failures --action_env=PYTHON_BIN_PATH=$PYTHONHOME \
77+
--action_env=PYTHON_LIB_PATH=$PYTHONPATH
78+
cp -f ./bazel-bin/src/bindings/_pydp.so ./src/pydp
79+
80+
- name: Build Google DP Windows
81+
if: runner.os == 'Windows'
82+
timeout-minutes: 20
83+
run: |
84+
$PYTHONHOME=$(python -c 'import sys; print(sys.executable);').replace('\', '/')
85+
$PYTHONPATH=$(python -c "import sys; print([x for x in sys.path if 'site-packages' in x][0]);").replace('\', '/')
86+
echo "PYTHONHOME=$PYTHONHOME"
87+
echo "PYTHONPATH=$PYTHONPATH"
88+
echo "Running: ${{ matrix.os }}"
89+
bazel.exe --output_base ./bazel-cache build src/python:pydp --config windows --verbose_failures --action_env=PYTHON_BIN_PATH=$PYTHONHOME --action_env=PYTHON_LIB_PATH=$PYTHONPATH
90+
copy ./bazel-bin/src/bindings/_pydp.so ./src/pydp/_pydp.pyd
91+
92+
- name: Upgrade pip
93+
run: |
94+
pip install --upgrade --user pip
95+
96+
- name: Install Poetry
97+
run: |
98+
pip install poetry
99+
100+
- name: Get poetry cache dir
101+
id: poetry-cache
102+
run: |
103+
echo "::set-output name=dir::$(poetry config cache-dir)"
104+
105+
- name: poetry cache
106+
uses: actions/cache@v2
107+
with:
108+
path: ${{ steps.poetry-cache.outputs.dir }}
109+
key: ${{ runner.os }}-pip-py${{ matrix.python-version }}-${{ hashFiles('**/pyproject.toml') }}
110+
restore-keys: |
111+
${{ runner.os }}-pip-py${{ matrix.python-version }}-
112+
113+
- name: Install dependencies
114+
run: |
115+
poetry install
116+
117+
- name: Build PyDP macOS
118+
if: runner.os == 'macOS'
119+
run: |
120+
poetry run python setup.py build bdist_wheel --plat-name macosx_10_14_x86_64
121+
122+
- name: Build PyDP Linux / Windows
123+
if: runner.os != 'macOS'
124+
run: |
125+
poetry run python setup.py build bdist_wheel
126+
127+
- name: Install Wheel Unix
128+
if: runner.os != 'Windows'
129+
run: |
130+
pip install `find -L ./ -name "*.whl"`
131+
132+
- name: Install Wheel Windows
133+
if: runner.os == 'Windows'
134+
run: |
135+
Get-ChildItem -Path ./ -Filter "*.whl" -Recurse -File | foreach {pip install $_.FullName}
136+
137+
- name: Import Package
138+
run: |
139+
python -c "import pydp; print(pydp.__version__)"
140+
141+
- name: Run Pytest
142+
run: |
143+
poetry run pytest tests -n auto
144+
145+
- name: Check Wheel Unix
146+
if: runner.os != 'Windows'
147+
run: |
148+
twine check `find -L ./ -name "*.whl"`
149+
150+
- name: Check Wheel Windows
151+
if: runner.os == 'Windows'
152+
run: |
153+
Get-ChildItem -Path ./ -Filter "*.whl" -Recurse -File | foreach {twine check $_.FullName}
154+
155+
- name: Renaming wheel
156+
if: runner.os == 'Linux'
157+
run: |
158+
find . -name '*linux*.whl' -type f -exec bash -c 'mv "$1" "${1/linux/manylinux1}"' -- {} \;
159+
160+
# - name: Publishing the wheel
161+
# env:
162+
# TWINE_USERNAME: __token__
163+
# TWINE_PASSWORD: ${{ secrets.TOKEN }}
164+
# run: |
165+
# twine upload --skip-existing dist/*.whl

0 commit comments

Comments
 (0)