LIBERO

🚀 LIBERO Evaluation

This document provides instructions for reproducing our experimental results with LIBERO.
The evaluation process consists of two main parts:

1. Setting up the LIBERO environment and dependencies.
2. Running the evaluation by launching services in both starVLA and LIBERO environments.

We have verified that this workflow runs successfully on both NVIDIA A100 and RTX 4090 GPUs.

💡 AMD GPU Support: Community members have verified that starVLA also works on AMD Instinct MI300X GPUs with ROCm 6.4 — with zero source code changes. The only modification needed is setting --framework.qwenvl.attn_implementation sdpa. For a detailed setup guide and benchmark results, see Issue #254.

---

⬇️ 0. Download Checkpoints

We provide a collection of pretrained checkpoints on Hugging Face to make community evaluation easier: 🤗 StarVLA/bench-libero. Their corresponding results on LIBERO are summarized in the table below.

📊 Experimental Results

Model	Steps	Epochs	Spatial	Object	Goal	Long	Avg
$\pi_0$+FAST	-	-	96.4	96.8	88.6	60.2	85.5
OpenVLA-OFT	175K	223	97.6	98.4	97.9	94.5	97.1
$\pi_0$	-	-	96.8	98.8	95.8	85.2	94.1
GR00T-N1.5	20K	203	92.0	92.0	86.0	76.0	86.5
StarVLA-FAST (Qwen2.5-VL)	30K	9.54	97.3	97.2	96.1	90.2	95.2
StarVLA-OFT (Qwen2.5-VL)	30K	9.54	97.4	98.0	96.8	92.0	96.1
StarVLA-π (Qwen2.5-VL)	30K	9.54	98.2	99.2	95.6	88.4	95.4
StarVLA-GR00T (Qwen2.5-VL)	30K	9.54	97.8	98.2	94.6	90.8	95.4
StarVLA-FAST (Qwen3-VL)	30K	9.54	97.3	97.4	96.3	90.6	95.4
StarVLA-OFT (Qwen3-VL)	30K	9.54	97.8	98.6	96.2	93.8	96.6
StarVLA-π (Qwen3-VL)	30K	9.54	98.8	99.6	95.8	88.4	95.7
StarVLA-GR00T (Qwen3-VL)	30K	9.54	97.8	98.8	97.4	92.0	96.5

We train one policy for all 4 suites. All scores are averaged over 500 trials for each task suite (10 tasks × 50 episodes).

---

📦 1. Environment Setup

To set up the environment, please first follow the official LIBERO repository to install the base LIBERO environment.

⚠️ Common issue: LIBERO defaults to Python 3.8, but the syntax updates between 3.8 and 3.10 are substantial. We verified that using Python 3.10 avoids many issues.

Afterwards, inside the LIBERO environment, install the following dependencies:

pip install tyro matplotlib mediapy websockets msgpack
pip install numpy==1.24.4 mujoco==3.2.3

---

🚀 2. Evaluation Workflow

The evaluation should be run from the repository root using two separate terminals, one for each environment:

• starVLA environment: runs the inference server.
• LIBERO environment: runs the simulation.

Step 1. Start the server (starVLA environment)

In the first terminal, activate the starVLA conda environment and run:

bash examples/simBenchmarks/LIBERO/eval_files/run_policy_server.sh

⚠️ Note: Please ensure that you specify the correct checkpoint path in examples/simBenchmarks/LIBERO/eval_files/run_policy_server.sh

Released Qwen3-PI LIBERO checkpoint compatibility

PR #373 introduced the canonical LayerwiseFM path used by current and newly trained models. The released StarVLA/Qwen3-VL-PI-LIBERO-4in1 checkpoint was trained with historical LayerwiseFM semantics, so evaluate that checkpoint with the compatibility override below:

CKPT=/path/to/StarVLA/Qwen3-VL-PI-LIBERO-4in1/checkpoints/steps_100000_pytorch_model.pt \
USE_CANONICAL_FORWARD=false \
bash examples/simBenchmarks/LIBERO/eval_files/run_policy_server.sh

When USE_CANONICAL_FORWARD is unset, the server keeps the canonical default. Do not edit the checkpoint config.yaml; use the launcher override instead. The released checkpoint config uses 4 inference steps.

---

Step 2. Start the simulation (LIBERO environment)

In the second terminal, activate the LIBERO conda environment and run:

bash examples/simBenchmarks/LIBERO/eval_files/eval_libero.sh

⚠️ Note: Please ensure that you specify the correct checkpoint path in eval_libero.sh to load action unnormalization stats.

Also ensure the environment variables at the top of eval_libero.sh are correctly set.

Finally, each result will also save a video for visualization, as shown below:

---

🚀 LIBERO Training

📦 Step 0: Download the training dataset

Download the datasets to the playground/Datasets/LEROBOT_LIBERO_DATA directory:

• LIBERO-spatial
• LIBERO-object
• LIBERO-goal
• LIBERO-10

And move modality.json to each $LEROBOT_LIBERO_DATA/subset/meta/modality.json.

You could quickly prepare these by running:

# Set DEST to the directory where you want to store the data
export DEST=/path/to/your/data/directory
bash examples/simBenchmarks/LIBERO/data_preparation.sh

🚀 Step1: Start Training

Most of the required training files have been organized in train_files.

Please run the following command to start training:

bash examples/simBenchmarks/LIBERO/train_files/run_libero_train.sh

⚠️ Note: Please ensure that you specify the correct path in examples/simBenchmarks/LIBERO/train_files/run_libero_train.sh