Code release

.gitignore

@@ -0,0 +1,179 @@
+.vscode
+.hydra
+inputs
+outputs
+
+# All file or folders start with tmp will be ignored
+tmp*  
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# 
+.DS_Store/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# torchsparse
+torchsparse
+
+# tensorboard
+tensorboard
+
+# glove
+glove

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "third-party/DPVO"]
+	path = third-party/DPVO
+	url = https://github.com/princeton-vl/DPVO.git

LICENSE

@@ -0,0 +1,16 @@
+Copyright 2022-2023 3D Vision Group at the State Key Lab of CAD&CG,  
+Zhejiang University. All Rights Reserved. 
+
+For more information see <https://github.com/zju3dv/GVHMR> 
+If you use this software, please cite the corresponding publications   
+listed on the above website. 
+
+Permission to use, copy, modify and distribute this software and its 
+documentation for educational, research and non-profit purposes only. 
+Any modification based on this work must be open-source and prohibited 
+for commercial use. 
+You must retain, in the source form of any derivative works that you  
+distribute, all copyright, patent, trademark, and attribution notices  
+from the source form of this work. 
+
+For commercial uses of this software, please send email to [email protected]

README.md

@@ -0,0 +1,63 @@
+# World-Grounded Human Motion Recovery via Gravity-View Coordinates
+### [Project Page](https://zju3dv.github.io/gvhmr) | [Paper](https://arxiv.org/pdf/xxxx.xxxxx.pdf)
+
+> World-Grounded Human Motion Recovery via Gravity-View Coordinates  
+> [Zehong Shen](https://zehongs.github.io/)<sup>\*</sup>, 
+[Huaijin Pi](https://phj128.github.io/)<sup>\*</sup>, 
+[Yan Xia](https://isshikihugh.github.io/scholar), 
+[Zhi Cen](https://scholar.google.com/citations?user=Xyy-uFMAAAAJ), 
+[Sida Peng](https://pengsida.net/)<sup>†</sup>, 
+[Zechen Hu](https://zju3dv.github.io/gvhmr), 
+[Hujun Bao](http://www.cad.zju.edu.cn/home/bao/), 
+[Ruizhen Hu](https://csse.szu.edu.cn/staff/ruizhenhu/), 
+[Xiaowei Zhou](https://xzhou.me/)  
+> Siggraph Asia 2024
+
+<p align="center">
+    <img src=docs/example_video/project_teaser.gif alt="animated" />
+</p>
+
+## TODO List and ETA
+- [x] Code for reproducing the train and test results (2024-8-5)
+- [x] Demo code (2024-8-5)
+- [x] Project page, Repository README (2024-9-4)
+- [ ] Arxiv paper link (~2024-9)
+
+
+## Setup
+
+Please see [installation](docs/INSTALL.md) for details.
+
+## Quick Start
+
+### Demo
+Demo entries are provided in `tools/demo`. Use `-s` to skip visual odometry if you know the camera is static, otherwise the camera will be estimated by DPVO.
+We also provide a script `demo_folder.py` to inference a entire folder. 
+```shell
+python tools/demo/demo.py --video=docs/example_video/tennis.mp4 -s
+python tools/demo/demo_folder.py -f inputs/demo/folder_in -d outputs/demo/folder_out -s
+```
+
+### Reproduce
+1. **Test**: 
+To reproduce the 3DPW, RICH, and EMDB results in a single run, use the following command:
+    ```shell
+    python tools/train.py global/task=gvhmr/test_3dpw_emdb_rich exp=gvhmr/mixed/mixed ckpt_path=inputs/checkpoints/gvhmr/gvhmr_siga24_release.ckpt
+    ```
+    To test individual datasets, change `global/task` to `gvhmr/test_3dpw`, `gvhmr/test_rich`, or `gvhmr/test_emdb`.
+
+2. **Train**:
+To train the model, use the following command:
+    ```shell
+    # The gvhmr_siga24_release.ckpt is trained with 2x4090 for 420 epochs, note that different GPU settings may lead to different results.
+    python tools/train.py exp=gvhmr/mixed/mixed
+    ```
+    During training, note that we do not employ post-processing as in the test script, so the global metrics results will differ (but should still be good for comparison with baseline methods).
+
+
+# Acknowledgement
+
+We thank the authors of 
+[WHAM](https://github.com/yohanshin/WHAM),
+[4D-Humans](https://github.com/shubham-goel/4D-Humans),
+and [ViTPose-Pytorch](https://github.com/gpastal24/ViTPose-Pytorch) for their great works, without which our project/code would not be possible.

docs/INSTALL.md

@@ -0,0 +1,88 @@
+# Install
+
+## Environment
+
+```bash
+git clone https://github.com/zju3dv/GVHMR --recursive
+cd GVHMR
+
+conda create -y -n gvhmr python=3.10
+conda activate gvhmr
+pip install -r requirements.txt
+pip install -e .
+# to install gvhmr in other repo as editable, try adding "python.analysis.extraPaths": ["path/to/your/package"] to settings.json
+
+# DPVO
+cd third-party/DPVO
+wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
+unzip eigen-3.4.0.zip -d thirdparty && rm -rf eigen-3.4.0.zip
+pip install torch-scatter -f "https://data.pyg.org/whl/torch-2.3.0+cu121.html"
+export CUDA_HOME=/usr/local/cuda-12.1/
+export PATH=$PATH:/usr/local/cuda-12.1/bin/
+pip install .
+```
+
+## Inputs & Outputs
+
+```bash
+mkdir inputs
+mkdir outputs
+```
+
+**Weights**
+
+```bash
+mkdir -p inputs/checkpoints
+
+# 1. You need to sign up for downloading [SMPL](https://smpl.is.tue.mpg.de/) and [SMPLX](https://smpl-x.is.tue.mpg.de/). And the checkpoints should be placed in the following structure:
+
+inputs/checkpoints/
+├── body_models/smplx/
+│   └── SMPLX_{GENDER}.npz # SMPLX (We predict SMPLX params + evaluation) 
+└── body_models/smpl/
+    └── SMPL_{GENDER}.pkl  # SMPL (rendering and evaluation)
+
+# 2. Download other pretrained models from Google-Drive (By downloading, you agree to the corresponding licences): https://drive.google.com/drive/folders/1eebJ13FUEXrKBawHpJroW0sNSxLjh9xD?usp=drive_link
+
+inputs/checkpoints/
+├── dpvo/
+│   └── dpvo.pth
+├── gvhmr/
+│   └── gvhmr_siga24_release.ckpt
+├── hmr2/
+│   └── epoch=10-step=25000.ckpt
+├── vitpose/
+│   └── vitpose-h-multi-coco.pth
+└── yolo/
+    └── yolov8x.pt
+```
+
+**Data**
+
+We provide preprocessed data for training and evaluation. 
+Note that we do not intend to distribute the original datasets, and you need to download them (annotation, videos, etc.) from the original websites.
+*We're unable to provide the original data due to the license restrictions.*
+By downloading the preprocessed data, you agree to the original dataset's terms of use and use the data for research purposes only. 
+
+You can download them from [Google-Drive](https://drive.google.com/drive/folders/10sEef1V_tULzddFxzCmDUpsIqfv7eP-P?usp=drive_link). Please place them in the "inputs" folder and execute the following commands:
+
+```bash
+cd inputs
+# Train
+tar -xzvf AMASS_hmr4d_support.tar.gz
+tar -xzvf BEDLAM_hmr4d_support.tar.gz
+tar -xzvf H36M_hmr4d_support.tar.gz
+# Test
+tar -xzvf 3DPW_hmr4d_support.tar.gz
+tar -xzvf EMDB_hmr4d_support.tar.gz
+tar -xzvf RICH_hmr4d_support.tar.gz
+
+# The folder structure should be like this:
+inputs/
+├── AMASS/hmr4d_support/
+├── BEDLAM/hmr4d_support/
+├── H36M/hmr4d_support/
+├── 3DPW/hmr4d_support/
+├── EMDB/hmr4d_support/
+└── RICH/hmr4d_support/
+```

hmr4d/__init__.py

@@ -0,0 +1,9 @@
+import os
+from pathlib import Path
+
+PROJ_ROOT = Path(__file__).resolve().parents[1]
+
+
+def os_chdir_to_proj_root():
+    """useful for running notebooks in different directories."""
+    os.chdir(PROJ_ROOT)

hmr4d/build_gvhmr.py

@@ -0,0 +1,11 @@
+from omegaconf import OmegaConf
+from hmr4d import PROJ_ROOT
+from hydra.utils import instantiate
+from hmr4d.model.gvhmr.gvhmr_pl_demo import DemoPL
+
+
+def build_gvhmr_demo():
+    cfg = OmegaConf.load(PROJ_ROOT / "hmr4d/configs/demo_gvhmr_model/siga24_release.yaml")
+    gvhmr_demo_pl: DemoPL = instantiate(cfg.model, _recursive_=False)
+    gvhmr_demo_pl.load_pretrained_model(PROJ_ROOT / "inputs/checkpoints/gvhmr/gvhmr_siga24_release.ckpt")
+    return gvhmr_demo_pl.eval()