This repo is the official implementation of "Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation".

📰 News

[2025/02/20] We release the DuoBox dataset, training and evaluation code.

🛠️ Installation

Conda environment

```bash conda create -n react python=3.9 conda activate react # install pytorch conda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 pytorch-cuda=11.6 -c pytorch -c nvidia # install pytroch3d pip install pytorch3d-0.7.2-cp39-cp39-linux_x86_64.whl # install other requirements cat requirements.txt | sed -e '/^\s*-.*$/d' -e '/^\s*#.*$/d' -e '/^\s*$/d' | awk '{split($0, a, "#"); if (length(a) > 1) print a[1]; else print $0;}' | awk '{split($0, a, "@"); if (length(a) > 1) print a[2]; else print $0;}' | xargs -n 1 pip install # install ready2react lib pip install -e . --no-build-isolation --no-deps ```

NOTE: pytorch3d download link.

💾 The DuoBox dataset

Download the DuoBox dataset

To download DuoBox, please fill in this form. We provide skeleton extracted from OptiTrack and SMPL-X fitting results. In this project, we use the skeleton extracted from OptiTrack. The skeleton definition can be found in link.

Dataset format

The data format of files under fbx_export_joints is: ```python { f'{sbj_name}_position': # (T, J, 3), global position of joints f'{sbj_name}_rotation': # (T, J, 3, 3), [:, 1] are the root global orientations, the [:, 1:] are the other joints' local rotations relative to their parents } ```

The data format of files under fbx_fit_smpl is:

{
    f'{sbj_name}':
    {
        'transl':  # (T, 3)
        'global_orient': # (T, 3)
        'body_pose': # (T, 63)
        'betas': # (T, 10)
    }
}

🔧 Preparation

Preprocess data

1. Link DuoBox to data/:

mkdir data
ln -s /path/to/DuoBox data/DuoBox

2. Dataset spliting and preparing for reactive motion model:

python reactmotion/scripts/main.py -t data_test -c configs/exps/vqvae_tokenizer.yaml dataloader_cfg.dataset_cfg.gen_motion_split=True
python reactmotion/scripts/main.py -t data_test -c configs/exps/vqvae_tokenizer.yaml dataloader_cfg.dataset_cfg.gen_preprocess=True
python reactmotion/scripts/main.py -t data_test -c configs/exps/reactive_model.yaml dataloader_cfg.dataset_cfg.gen_preprocess=True

Files will be saved in data/boxing/reactive.

3. Dataset spliting and preparing for sparse control model:

python reactmotion/scripts/main.py -t data_test -c configs/exps/sparse_control.yaml dataloader_cfg.dataset_cfg.gen_motion_split=True
python reactmotion/scripts/main.py -t data_test -c configs/exps/sparse_control.yaml dataloader_cfg.dataset_cfg.gen_preprocess=True

Download pretrained models

Weights are shared in link. Download and unzip under data/.

/data
|__ DuoBox
|__ trained_model

🚀 Usage

1️⃣ Reactive motion generation

python reactmotion/scripts/main.py -t test -c configs/exps/reactive_model.yaml model_cfg.inference_func=reactive

2️⃣ Two-character motion generation

python reactmotion/scripts/main.py -t test -c configs/exps/reactive_model.yaml model_cfg.inference_func=twoagent

3️⃣ Long-term two-character motion generation

python reactmotion/scripts/main.py -t test -c configs/exps/reactive_model.yaml model_cfg.inference_func=twoagent model_cfg.use_gt_length=False model_cfg.num_new_frames=1800

4️⃣ Sparse control

python reactmotion/scripts/main.py -t test -c configs/exps/sparse_control.yaml model_cfg.inference_func=reactive

Useful flags

model_cfg.inference_func=reactive or twoagent # inference type
runner_cfg.evaluator_cfg.feature_compute=False # skip eval FID
runner_cfg.evaluator_cfg.apd_compute=True # eval APD

🧠 Training the model by yourself

To train the models, simply set -t train istead of -t test. Don't forget to change the exp_name=yourexpname.

1️⃣ Train the VQ-VAE motion tokenizer

python reactmotion/scripts/main.py -t train -c configs/exps/vqvae_tokenizer.yaml exp_name=yourexpname1

2️⃣ Train the reactive motion model

Set the config motoken_cfg_file to the config file generated in step # 1. It should be saved in data/record/yourexpname1.

python reactmotion/scripts/main.py -t train -c configs/exps/reactive_model.yaml exp_name=yourexpname2

🙏 Acknowledgements

• EasyVolCap for their nice code base.
• T2M-GPT for the VQ-VAE architecture.

📚 Citation

If you use our model or dataset, please cite our Ready-to-React paper. If you use easyvolcap codebase, please cite EasyVolcap as follows.

@inproceedings{cen2025_ready_to_react,
  title={Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation},
  author={Cen, Zhi and Pi, Huaijin and Peng, Sida and Shuai, Qing and Shen, Yujun and Bao, Hujun and Zhou, Xiaowei and Hu, Ruizhen},
  booktitle={ICLR},
  year={2025}
}
@article{xu2023easyvolcap,
  title={EasyVolcap: Accelerating Neural Volumetric Video Research},
  author={Xu, Zhen and Xie, Tao and Peng, Sida and Lin, Haotong and Shuai, Qing and Yu, Zhiyuan and He, Guangzhao and Sun, Jiaming and Bao, Hujun and Zhou, Xiaowei},
  booktitle={SIGGRAPH Asia 2023 Technical Communications},
  year={2023}
}