Create environment
Create conda environment:
conda create --name director python=3.10 -y
conda activate director
Install dependencies:
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=12.1 -c pytorch -c nvidia
conda install pytorch3d -c pytorch3d
pip install -r requirements.txt
Note: pytorch3d installation can be tricky. Please follow the official guidelines if you encounter any issues.
Set up the dataset
Check the data repository here.
First, install git lfs by following the instructions here.
To get the data, run:
git clone https://huggingface.co/datasets/robin-courant/pulpmotion-data
Prepare the dataset (untar archives):
cd pulpmotion-data
sh untar.sh
cd ..
Note: Model checkpoints are stored in the runs directory. Checkpoints trained on the mixed subsets correspond to 330,750 training steps, while those trained on the pure subsets correspond to 92,950 training steps. Checkpoints names follow: {arch}-{mode}-ddpm-{id}-{num_steps}.ckpt.
Download checkpoints
Check the model repository here.
First, install git lfs by following the instructions here.
To get the data, run:
git clone https://huggingface.co/robin-courant/pulpmotion-models
Register at SMPLify and SMPL, whose usernames and passwords will be used to download the SMPL models.
cd pulpmotion-models
sh download_smpl.sh
cd ..
Config name details
| Arch | Mode | Description | CONFIG_NAME |
|---|---|---|---|
dit |
x_y |
DiT with independent modalities for x and y (split 2D streams) | config_dit_x_y.yaml |
dit |
xy |
DiT with dual-modalities for joint 2D xy | config_dit_xy.yaml |
dit |
xyz |
DiT with triplet modalities for full 3D | config_dit_xyz.yaml |
mar |
x_y |
MAR with independent x and y | config_mar_x_y.yaml |
mar |
xy |
MAR with dual-modalities for joint 2D xy | config_mar_xy.yaml |
mar |
xyz |
MAR with triplet modalities for full 3D | config_mar_xyz.yaml |
To launch training, run:
HYDRA_FULL_ERROR=1 python src/train.py --config-name=CONFIG_NAME seed=42 metrics.camera.do_segments=false
To launch evaluation, run:
HYDRA_FULL_ERROR=1 python src/evaluate.py --config-name=CONFIG_NAME checkpoint_path=CHECKPOINT_PATH compnode.num_gpus=1 \
model.sampler.generation_sampler.cfg_rate_c=TEXT_WEIGHT model.sampler.generation_sampler.cfg_rate_z=FRAMING_WEIGHT
This code is distributed under an MIT LICENSE.
Note that our code depends on other libraries, including PyTorch, PyTorch3D, Hugging Face, Hydra, and uses datasets which each have their own respective licenses that must also be followed.