Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI nightlies cpu/gpu & cleanup #75

Merged
merged 50 commits into from
Apr 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
547348a
Improve dataset examples (#82)
Cadene Apr 18, 2024
c5ebf23
Update lock after rebase main
aliberts Apr 23, 2024
55ffb42
Add install EGL
aliberts Apr 23, 2024
8ea29b2
remove sudo
aliberts Apr 23, 2024
45443d4
Try display :0
aliberts Apr 23, 2024
3896786
MUJOCO_EGL_DEVICE_ID=0
aliberts Apr 23, 2024
c1f3e67
unset DISPLAY
aliberts Apr 23, 2024
3e68c00
Trying xvfb
aliberts Apr 23, 2024
5fc1bcb
Run with new entrypoint
aliberts Apr 23, 2024
0a86538
Pass global env variables
aliberts Apr 24, 2024
f4ed7a5
add ssh tailscale
glegendre01 Apr 24, 2024
6f331d1
add curl + privileged
glegendre01 Apr 24, 2024
9dbccdb
add ca-certificates
glegendre01 Apr 24, 2024
60b171e
change TS action to V1
glegendre01 Apr 24, 2024
96bca57
Try internal base image
aliberts Apr 24, 2024
530ef06
Try nvidia/cuda:12.4.1-base-ubuntu22.04
aliberts Apr 24, 2024
050fa00
Trigger issue 1/2
aliberts Apr 24, 2024
d5fbabd
Trigger issue 1/2
aliberts Apr 24, 2024
467989a
Trigger issue 2/2
aliberts Apr 24, 2024
9f1d2a6
Test new image
aliberts Apr 24, 2024
433dfce
Fix trigger
aliberts Apr 24, 2024
b910170
Push new build
aliberts Apr 24, 2024
86a2021
Try new build
aliberts Apr 24, 2024
7a9b92d
Thinning image
aliberts Apr 24, 2024
35c1877
Cleanup & test thinner image
aliberts Apr 24, 2024
b836883
Thinning image more
aliberts Apr 24, 2024
0642f42
Testing thinned image
aliberts Apr 24, 2024
5ea2d2c
Thinning image more
aliberts Apr 24, 2024
88dae64
Testing thinned image
aliberts Apr 24, 2024
7da5df8
Fixing & thinning
aliberts Apr 24, 2024
a6add57
Testing thinned image
aliberts Apr 24, 2024
4f0048c
Adding back libglib2.0-0
aliberts Apr 24, 2024
e8bed61
Testing with libglib2.0-0
aliberts Apr 24, 2024
ffc178a
Fixing
aliberts Apr 24, 2024
cde7f0d
Fixing
aliberts Apr 24, 2024
d4eeaa6
Print device in tests
aliberts Apr 25, 2024
10755a9
Cleaning cpu & gpu images
aliberts Apr 25, 2024
4144f71
Build new images
aliberts Apr 25, 2024
d0d64bf
Fix python version
aliberts Apr 25, 2024
aa8025a
Test builds
aliberts Apr 25, 2024
35248a9
Change jobs names
aliberts Apr 25, 2024
5fda961
Trying new cache strategy
aliberts Apr 25, 2024
ec4b988
Cleanup
aliberts Apr 25, 2024
5cb4799
More cleanup
aliberts Apr 25, 2024
e2b196b
Fix poetry files path
aliberts Apr 25, 2024
9afd0ed
Fix poetry files path (cpu)
aliberts Apr 25, 2024
3416341
Use script to change pyproject instead of 2 versions
aliberts Apr 25, 2024
67c3554
Last test
aliberts Apr 25, 2024
c2fcfd8
Cleanup
aliberts Apr 25, 2024
9373d36
More explicit end-to-end test name
aliberts Apr 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 142 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# Misc
.git
tmp
wandb
data
outputs
.vscode
rl
media


# Logging
logs

# HPC
nautilus/*.yaml
*.key

# Slurm
sbatch*.sh

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
!tests/data
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/
25 changes: 21 additions & 4 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,29 @@
# What does this PR do?

Example: Fixes # (issue)
Examples:
- Fixes # (issue)
- Adds new dataset
- Optimizes something

## How was it tested?

Examples:
- Added `test_something` in `tests/test_stuff.py`.
- Added `new_feature` and checked that training converges with policy X on dataset/environment Y.
- Optimized `some_function`, it now runs X times faster than previously.

## How to checkout & try? (for the reviewer)

Examples:
```bash
DATA_DIR=tests/data pytest -sx tests/test_stuff.py::test_something
```
```bash
python lerobot/scripts/train.py --some.option=true
```

## Before submitting
- Read the [contributor guideline](https://github.com/huggingface/lerobot/blob/main/CONTRIBUTING.md#submitting-a-pull-request-pr).
- Provide a minimal code example for the reviewer to checkout & try.
- Explain how you tested your changes.
Please read the [contributor guideline](https://github.com/huggingface/lerobot/blob/main/CONTRIBUTING.md#submitting-a-pull-request-pr).


## Who can review?
Expand Down
30 changes: 30 additions & 0 deletions .github/scripts/dep_build.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
PYPROJECT = "pyproject.toml"
DEPS = {
"gym-pusht": '{ git = "[email protected]:huggingface/gym-pusht.git", optional = true}',
"gym-xarm": '{ git = "[email protected]:huggingface/gym-xarm.git", optional = true}',
"gym-aloha": '{ git = "[email protected]:huggingface/gym-aloha.git", optional = true}',
}


def update_envs_as_path_dependencies():
with open(PYPROJECT) as file:
lines = file.readlines()

new_lines = []
for line in lines:
if any(dep in line for dep in DEPS.values()):
for dep in DEPS:
if dep in line:
new_line = f'{dep} = {{ path = "envs/{dep}/", optional = true}}\n'
new_lines.append(new_line)
break

else:
new_lines.append(line)

with open(PYPROJECT, "w") as file:
file.writelines(new_lines)


if __name__ == "__main__":
update_envs_as_path_dependencies()
Loading