Bingquan Dai*, Li Ray Luo*, Qihong Tang,
Jie Wang, Xinyu Lian, Hao Xu, Minghan Qin, Xudong Xu, Bo Dai, Haoqian Wang†, Zhaoyang Lyu† Jiangmiao Pang
* Equal contribution
† Corresponding author
Project lead: Zhaoyang Lyu
[2025.11.11] We release the training and inference code, the pretrained checkpoint of the object-to-code model, and 100K training data of paired object and code.
MeshCoder is a framework that converts 3D point clouds into editable Blender Python scripts, enabling programmatic reconstruction and editing of complex human-made objects. It overcomes prior limitations by developing expressive APIs for modeling intricate geometries, building a large-scale dataset of 1 million object-code pairs across 41 categories, and training a multimodal LLM to generate accurate, part-segmented code from point clouds. The approach outperforms existing methods in reconstruction quality, supports intuitive shape and topology editing via code modifications, and enhances 3D reasoning capabilities in LLMs.
Our code is tested with Python 3.10, PyTorch 2.4.1 and CUDA 11.8
- Clone MeshCoder
git clone https://github.com/InternRobotics/MeshCoder.git
cd MeshCoder
- Install relevant packages using conda and pip
conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=11.8 -c pytorch -c nvidia -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d -c pytorch3d
conda install typeguard
pip install torch-scatter -f https://data.pyg.org/whl/torch-2.4.1+cu118.html
pip install bpy==4.0.0
pip install opencv-python trimesh jaxtyping einops omegaconf scikit-image wandb mathutils boto3
pip install -U xformers==0.0.28 --index-url https://download.pytorch.org/whl/cu118 --no-deps
pip install -e .
You may mannually download bpy 4.0.0 and run pip install bpy-4.0.0-cp310-cp310-manylinux_2_28_x86_64.whl if pip can not install it properly.
Download the pretrained models from Hugging Face.
Move the shape tokenizer and the LoRA weight (config.yaml,shape_tokenizer.pt,adapter_config.json and adapter_model.safetensors) to the folder MeshCoder/llama-3-models/object_to_code_model.
Move the original Llama-3.2-1B model to MeshCoder/llama-3-models/Llama3.2-1B.
We provide an npz file containing point clouds generated by Infinigen (in-distribution) here, and an npz file containing point clouds generated by TRELLIS (out-of-distribution) here.
You can download it and put it under data folder.
Or you can prepare your own npz file which is organized as following.
data.npz
--points //point cloud of the shape you want to reconstruct, normalized to -1 - 1.
--normals //normals of the point cloud \
Run the following command to performance inference on given point clouds:
cd recipes/inference/local_inference/custom_inference
python inference_shape2code_point_input.py \
--model_name MeshCoder/llama-3-models/Llama3.2-1B \
--peft_model MeshCoder/llama-3-models/object_to_code_model \
--file_start_idx 0 --file_end_idx 10000000 \
--npz_data_file 'data/sample.npz' \
--code_save_file 'res/code.py' \
--skip_existing_files True \
--batch_size 8 \
--visualization \
--do_sample False \
--max_new_tokens 2048 \
--zero_out_normals True
Setup the environment variables NODE_RANK, MASTER_ADDR, MASTER_PORT according to your inference platform. Below is an example to inference the model with 4*8=32 GPUs.
cd recipes/inference/local_inference/custom_inference
torchrun \
--nnodes=4 \
--nproc_per_node=8 \
--node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR \
--master_port=$MASTER_PORT \
inference_shape2code_point_input.py \
--model_name MeshCoder/llama-3-models/Llama3.2-1B \
--peft_model MeshCoder/llama-3-models/object_to_code_model \
--file_start_idx 0 --file_end_idx 10000000 \
--npz_data_file 'data/sample.npz' \
--code_save_file 'res/code.py' \
--skip_existing_files True \
--batch_size 8 \
--visualization \
--do_sample False \
--max_new_tokens 2048 \
--zero_out_normals True
If you want to infer several npz files containing point clouds, you can put all npz files path to a txt file and their correspoding save paths into another txt file.
Replace the code_save_file and npz_data_file with paths to your newly created txt files like this:
--npz_data_file 'data/all_npzs.txt' \
--code_save_file 'res/all_save_paths.txt' \
You can also infer only a fraction of npz files in your txt files by setting file_start_idx and file_end_idx which specify the range of files to be infered.
--npz_data_file 'data/all_npzs.txt' \
--code_save_file 'res/all_save_paths.txt' \
MeshCoder is capable of reconstructing certain samples within the out-of-distribution (OOD) data generated by TRELLIS, thereby validating its generalization capability. However, the model is unable to faithfully reconstruct some samples when the point cloud distribution is markedly deviant from that of its training data. In future research, we will endeavor to further enhance the generalization ability of MeshCoder.
Download the MeshCoder dataset.
Move files train.json, val.json, test.json and folders dataset_train, dataset_val to MeshCoder/src/llama_recipes/datasets/shape2code_dataset/object2code/MeshCoderData.
Unzip the point cloud files in folders dataset_train, dataset_val.
Run the following command to train the shape tokenizer and LoRA weight from scratch:
python -m llama_recipes.finetuning_shape2code \
--model_name llama-3-models/Llama3.2-1B \
--use_peft \
--peft_method lora \
--lora_config.r 16 \
--shape_tokenizer_config llama-3-models/object_to_code_model/config.yaml \
--output_dir llama-3-models/object_to_code_model_exp_1 \
--dataset_version simplify_dataset \
--max_sequence_length 5000 --max_cd_loss 0.005 \
--batching_strategy padding \
--gradient_accumulation_steps 4 \
--batch_size_training 2 \
--num_workers_dataloader 8 \
--val_batch_size 4 \
--batch_sampler None \
--num_epochs 10 \
--shape_tokenizer_lr_scheduler_step_size 100 \
--lr_scheduler_step_size 100 \
--lr 1e-4 --shape_tokenizer_lr 1e-4 \
--dataset shape2code_dataset --data_dir 'object2code/MeshCoderData' \
--num_proc 32 \
--num_samples 'train: -1; validation: -1; test: -1' \
--dataset_labels 'infinigen' \
--visualization True
If you want to finetune the shape tokenizer and the LoRA weight from our provided checkpoint, add the following args to the above command.
--resume_training_dir llama-3-models/object_to_code_model --load_optimizer_state_dict
Setup the environment variables NODE_RANK, MASTER_ADDR, MASTER_PORT according to your training platform. Below is an example to train the model with 4*8=32 GPUs.
torchrun \
--nnodes=4 \
--nproc_per_node=8 \
--node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR \
--master_port=$MASTER_PORT \
-m llama_recipes.finetuning_shape2code \
--enable_fsdp \
--model_name llama-3-models/Llama3.2-1B \
--use_peft \
--peft_method lora \
--lora_config.r 16 \
--shape_tokenizer_config llama-3-models/object_to_code_model/config.yaml \
--output_dir llama-3-models/object_to_code_model_exp_1 \
--dataset_version simplify_dataset \
--max_sequence_length 5000 --max_cd_loss 0.005 \
--batching_strategy padding \
--gradient_accumulation_steps 4 \
--batch_size_training 2 \
--num_workers_dataloader 8 \
--val_batch_size 4 \
--batch_sampler None \
--num_epochs 10 \
--shape_tokenizer_lr_scheduler_step_size 100 \
--lr_scheduler_step_size 100 \
--lr 1e-4 --shape_tokenizer_lr 1e-4 \
--dataset shape2code_dataset --data_dir 'object2code/MeshCoderData' \
--num_proc 32 \
--num_samples 'train: -1; validation: -1; test: -1' \
--dataset_labels 'infinigen' \
--visualization True
If you want to finetune the shape tokenizer and the LoRA weight from our provided checkpoint, add the following args to the above coommand.
--resume_training_dir llama-3-models/object_to_code_model --load_optimizer_state_dict
We are seeking engineers, interns, researchers, and PhD candidates. If you have an interest in 3D content generation, please send your resume to [email protected].
If you find our work helpful, please consider citing:
@article{dai2025meshcoder,
title={MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds},
author={Dai, Bingquan and Luo, Li Ray and Tang, Qihong and Wang, Jie and Lian, Xinyu and Xu, Hao and Qin, Minghan and Xu, Xudong and Dai, Bo and Wang, Haoqian and others},
journal={arXiv preprint arXiv:2508.14879},
year={2025}
}
We thank all authors behind these repositories for their excellent work: Llama, TripoSR.
