DepthSplat: Connecting Gaussian Splatting and Depth

Haofei Xu · Songyou Peng · Fangjinhua Wang · Hermann Blum · Daniel Barath · Andreas Geiger · Marc Pollefeys

Paper | Project Page

DepthSplat enables cross-task interactions between Gaussian splatting and depth estimation.

Installation

Our code is developed based on pytorch 2.4.0, CUDA 12.4 and python 3.10.

We recommend using conda for installation:

conda create -y -n depthsplat python=3.10
conda activate depthsplat

pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu124
pip install xformers==0.0.27.post2
pip install -r requirements.txt

Model Zoo

Our models are hosted on Hugging Face 🤗 : https://huggingface.co/haofeixu/depthsplat

Model details can be found at MODEL_ZOO.md.

We assume the downloaded weights are located in the pretrained directory.

Camera Conventions

The camera intrinsic matrices are normalized (the first row is divided by image width, and the second row is divided by image height).

The camera extrinsic matrices are OpenCV-style camera-to-world matrices ( +X right, +Y down, +Z camera looks into the screen).

Datasets

Please refer to DATASETS.md for dataset preparation.

Depth Prediction

Please check scripts/inference_depth_small.sh, scripts/inference_depth_base.sh, and scripts/inference_depth_large.sh for scale-consistent depth prediction with models of different sizes.

We plan to release a simple depth inference pipeline in UniMatch repo.

Gaussian Splatting

• The training, evaluation, and rendering scripts on RealEstate10K dataset are available at scripts/re10k_256x256_depthsplat_small.sh, scripts/re10k_256x256_depthsplat_base.sh, and scripts/re10k_256x256_depthsplat_large.sh.

• The training, evaluation, and rendering scripts on DL3DV dataset are available at scripts/dl3dv_256x448_depthsplat_base.sh.

• Before training, you need to download the pre-trained Depth Anything V2 and UniMatch weights and set up your wandb account for logging.

wget https://huggingface.co/depth-anything/Depth-Anything-V2-Small/resolve/main/depth_anything_v2_vits.pth -P pretrained
wget https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale1-things-e9887eda.pth -P pretrained

Citation

@article{xu2024depthsplat,
      title   = {DepthSplat: Connecting Gaussian Splatting and Depth},
      author  = {Xu, Haofei and Peng, Songyou and Wang, Fangjinhua and Blum, Hermann and Barath, Daniel and Geiger, Andreas and Pollefeys, Marc},
      journal = {arXiv preprint arXiv:2410.13862},
      year    = {2024}
    }

Acknowledgements

This project is developed with several fantastic repos: pixelSplat, MVSplat, UniMatch, Depth Anything V2 and DL3DV. We thank the original authors for their excellent work.