MA-HMR

This repository contains the official implementation of Metric-Aware HMR for the paper Towards Metric-Aware Multi-Person Mesh Recovery by Jointly Optimizing Human Crowd in Camera Space. The official implementation of DTO framework and DTO-Humans dataset is available at DTO-Humans

Overview of MA-HMR

We propose Metric-Aware HMR, an end-to-end network that directly estimates human mesh and camera parameters in metric scale. This is enabled by a camera branch and a novel relative metric loss that enforces plausible relative scales.

⚙️ Installation

We follow SAT-HMR, testing with python 3.11, PyTorch 2.4.1 and CUDA 12.1.

1. Clone the repo and create a conda environment.

conda create -n mahmr python=3.11 -y
conda activate mahmr

2. Install PyTorch and xFormers.

# Install PyTorch. It is recommended that you follow [official instruction](https://pytorch.org/) and adapt the cuda version to yours.
conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=12.1 -c pytorch -c nvidia

# Install xFormers. It is recommended that you follow [official instruction](https://github.com/facebookresearch/xformers) and adapt the cuda version to yours.
pip install -U xformers==0.0.28.post1  --index-url https://download.pytorch.org/whl/cu121

3. Install other dependencies.

pip install -r requirements.txt

4. You may need to modify chumpy package to avoid errors. For detailed instructions, please check this guidance.

📦 Download Models & Weights

1. Download SMPL-related weights and place them in weights/smpl_data/smpl/. Partially Available at this link. You need to register on the SMPL website to get other part of them.

weights/
└── smpl_data/
    └── smpl/
        ├── body_verts_smpl.npy
        ├── J_regressor_h36m_correct.npy
        ├── J_regressor_extra.npy
        ├── smpl_mean_params.npz
        ├── SMPL_FEMALE.pkl
        ├── SMPL_MALE.pkl
        ├── SMPL_NEUTRAL.pkl
        └── smpl_kid_template.npy

2. Download DINOv2 pretrained weights from their official repository. We use ViT-B/14 distilled (without registers). Please put dinov2_vitb14_pretrain.pth to weights/dinov2. These weights will be used to initialize our encoder. You can skip this step if you are not going to train MA-HMR.

weights/
└── dinov2
    └── dinov2_vitb14_pretrain.pth

3. Download pretrained weights of SAT-HMR stage1 from Google drive or 🤗HuggingFace. Please put them to weights/sat_hmr. You can skip this step if you are not going to train MA-HMR.

weights
└── sat_hmr
    └── sat_644.pth

4. Download pretrained weights of MA-HMR from Google Drive | Tsinghua Cloud and put them to weights/ma_hmr. You can only prepare mahmr_stage3.bin for inference or evaluation.

• mahmr_stage1.bin: Start from SAT-HMR stage1, continue training on AGORA, BEDLAM and CameraHMR's version of 4D-humans pGTs (supervision on 2D kps) for 5 epochs with denoising strategy.
• mahmr_stage2.bin: Start from mahmr_stage1, continue training on AGORA, BEDLAM and CameraHMR's version of 4D-humans pGTs (supervision on 2D kps) for 5 epochs with camera branch and fov loss.
• mahmr_stage3.bin: Start from mahmr_stage2, continue training on AGORA, BEDLAM and DTO-Humans for 5 epochs with camera branch and full loss.

weights
└── ma_hmr
    ├── mahmr_stage1.bin
    ├── mahmr_stage2.bin
    └── mahmr_stage3.bin

📦 Data Preparation

Please follow this guidance to prepare AGORA, BEDLAM and 3DPW. Please refer to scripts in datasets/preprocess/ to preprocess HI4D, MuPoTS and CameraHMR's annotation of 4D-humans dataset. Download DTO-Humans annotations from Google Drive | Tsinghua Cloud. Placing all datasets in data/. You can skip this step if you are not going to train or evaluate MA-HMR.

data/
├── 3dpw/
│   ├── imageFiles/
│   ├── annots_smpl_test_genders.npz
│   └── annots_smpl_train_genders.npz
├── agora
│   ├── smpl_neutral_annots
│   │   ├── annots_smpl_train_fit.npz
│   │   └── annots_smpl_validation.npz
│   ├── test/
│   ├── train/
│   └── validation/
├── aic/
│   ├── images/
│   ├── aic-release.npz
│   ├── AIC_CHMR_SMPL.npz
│   └── AIC_CHMR_SMPL_OPT.npz
├── bedlam
│   ├── train/
│   ├── validation/
│   ├── bedlam_smpl_train_1fps.npz
│   ├── bedlam_smpl_train_6fps.npz
│   └── bedlam_smpl_validation_6fps.npz
├── cmu_panoptic/
│   ├── images/
│   └── annots_test.npz
├── coco2014/
│   ├── images/
│   │   └── train2014/
│   ├── coco-release.npz
│   ├── COCO_CHMR_SMPL.npz
│   └── COCO_CHMR_SMPL_OPT.npz
├── hi4d/
│   ├── pair**/
│   ├── hi4d_smpl_test.npz
│   └── hi4d_smpl_train.npz
├── insta/
│   ├── images/
│   │   └── insta-train/
│   ├── insta1-release.npz
│   ├── insta2-release.npz
│   ├── INSTA_CHMR_SMPL.npz
│   └── INSTA_CHMR_SMPL_OPT.npz
├── mpii/
│   ├── images/
│   ├── mpii-release.npz
│   ├── MPII_CHMR_SMPL.npz
│   └── MPII_CHMR_SMPL_OPT.npz
├── mupots/
│   ├── MultiPersonTestSet/
│   └── mupots_annots.npz
└── RelativeHuman/
    ├── images/
    ├── test_annots.npz
    └── train_annots.npz

▶️ Inference on Images

Inference with 1 GPU

We provide some demo images in demo/. You can run MA-HMR on all images on a single GPU via:

python main.py --mode infer --cfg demo

Results with overlayed meshes will be saved in ${Project}/demo_results.

You can specify your own inference configuration by modifing configs/run/demo.yaml:

• input_dir specifies the input image folder.
• output_dir specifies the output folder.
• conf_thresh specifies a list of confidence thresholds used for detection. SAT-HMR will run inference using thresholds in the list, respectively.
• infer_batch_size specifies the batch size used for inference (on a single GPU).

Inference with Multiple GPUs

You can also try distributed inference on multiple GPUs if your input folder contains a large number of images. Since we use Accelerate to launch our distributed configuration, first you may need to configure Accelerate for how the current system is setup for distributed process. To do so run the following command and answer the questions prompted to you:

accelerate config

Then run:

accelerate launch main.py --mode infer --cfg demo

🔧 Training

Training with Multiple GPUs

We use Accelerate to launch our distributed configuration, first you may need to configure Accelerate for how the current system is setup for distributed process. To do so run the following command and answer the questions prompted to you:

accelerate config

# stage 1
accelerate launch main.py --mode train --cfg train_chmr

# stage 2
accelerate launch main.py --mode train --cfg train_chmr_fov_continue

# stage 3
accelerate launch main.py --mode train --cfg train_chmr_fov_opt_mloss

# finetune
accelerate launch main.py --mode train --cfg 3dpw_ft

Monitor Training Progress

Training logs and checkpoints will be saved in the ${Project}/outputs/logs and ${Project}/outputs/ckpts directories, respectively.

You can monitor the training progress using TensorBoard. To start TensorBoard, run:

tensorboard --logdir=${Project}/outputs/logs

📊 Evaluation

Evaluation with 1 GPU

# Evaluate on rh
python main.py --mode eval --cfg eval_rh

Evaluation with Multiple GPUs

We recommend using a single GPU for evaluation as it provides more accurate results. However, we also provide code for distributed evaluation to obtain results faster.

# Multi-GPU configuration
accelerate config
# Evaluation
accelerate launch main.py --mode eval --cfg ${cfg_name}

📜 License

The code and weights are released under the Creative Commons Attribution-NonCommercial 4.0 International License. This means they are available for non-commercial academic research purposes only. Please see the LICENSE file for the full license text.

📜 Citation

If you find our work useful, please consider citing our paper:

@article{wang2025dtohumans,
  title={Towards Metric-Aware Multi-Person Mesh Recovery by Jointly Optimizing Human Crowd in Camera Space},
  author={Kaiwen Wang, Kaili Zheng, Yiming Shi, Chenyi Guo, Ji Wu},
  journal={arXiv preprint arXiv:2511.13282},
  year={2025}
}

🙏 Acknowledgements

This project builds upon several amazing open-source projects and datasets. We would like to thank the authors of:

• SAT-HMR
• CameraHMR
• DINOv2
• Accelerate