An Implementation of Variational Mapper on SDS based on ThreeStudio

Implemented based on LODS and ThreeStudio

LODS

ThreeStudio

Updates

• 29/2/2024: Code Released.

TODO List

• Implement improved SDS Loss (Variational Mapper)

• Apply improved SDS Loss on 3D Avatar Generation

• ID Driven 3D Avatar Generation

Installation

• Install PyTorch and torch vision

# torch1.12.1+cu113
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
# or torch2.0.0+cu118
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

• Install dependencies:

pip install -r requirements.txt

• Install 3DGS and Shap-E:

pip install ninja
pip install ./gaussiansplatting/submodules/diff-gaussian-rasterization
pip install ./gaussiansplatting/submodules/simple-knn

git clone https://github.com/openai/shap-e.git
cd shap-e
pip install -e .

If you have any problem with the installation, feel free to open a new issue here.

Quickstart

Run Variational Mapper + 3D Gaussian Splatting

python launch.py --config configs/t3aga-gs-mapper.yaml --train --gpu 0 system.prompt_processor.prompt="a DSLR image of a hamburger"

Credits

This code is built on the following open-source projects:

• ThreeStudio Main Framework
• LODS Clear Framework based on ThreeStudio
• GaussianDreamer 3D Gaussian Splatting
• 3D Gaussian Splatting 3D Gaussian Splatting

Credits from ThreeStudio

• Lightning Framework for creating highly organized PyTorch code.
• OmegaConf Flexible Python configuration system.
• NerfAcc Plug-and-play NeRF acceleration.