---

Veritas

🌐 Website

Visual Verification Enables Inference-time Steering and Autonomous Policy Improvement is a generator–verifier framework for generalist robot policies. A pre-trained policy proposes candidate action chunks; a gradient-free visual verifier scores them in pixel space against a Gemini-emitted waypoint plan and steers execution to the best one at inference time. The same verified rollouts can be recycled as new training data, enabling continual policy self-improvement without human interventions.

Installation

Clone the repository (recursively)

git clone --recursive git@github.com:robo-alex/veritas.git
cd veritas

If you cloned without --recursive:

git submodule update --init --recursive

Setup environment with uv

See installation instructions.

We use CUDA 13 by default. If you have an another version of CUDA installed, please change the torch / torchvision versions and index URL to be compatible in the pyproject.toml.

uv sync

Download PaliGemma weights

export HF_HUB_CACHE=$PWD/transformers_cache
mkdir -p $HF_HUB_CACHE
git clone https://huggingface.co/google/paligemma-3b-pt-224 $HF_HUB_CACHE/paligemma-3b-pt-224

Download a SAM2 + Grounding-DINO checkpoint

The Veritas verifier uses SAM2 hiera_large + Grounding-DINO base for object localization. From the repo root:

bash Grounded-SAM-2/checkpoints/download_ckpts.sh

Download the open-pi-zero policy checkpoint

Pull the Bridge policy from HuggingFace into models/.

mkdir -p models
uvx hf download allenzren/open-pi-zero \
  bridge_uniform_step19296_2024-12-26_22-31_42.pt \
  --local-dir models

Getting Started

Minimal code example

A raw rollout with no verifier — useful to confirm the env, checkpoint, and venv are wired up:

import hydra
from omegaconf import OmegaConf
from src.agent.eval import EvalAgent

cfg = OmegaConf.load("config/eval/bridge.yaml")
cfg.env.task = "widowx_put_eggplant_in_basket"
cfg.checkpoint_path = "models/bridge_uniform_step19296_2024-12-26_22-31_42.pt"
cfg.use_veritas_verifier = False
cfg.verifier_samples = 1
cfg.log_dir = "logs/raw/eval_widowx_put_eggplant_in_basket"

agent = EvalAgent(cfg)
agent.run()

Run the Veritas verifier on all WidowX tasks

export GEMINI_API_KEY=...
bash eval_with_veritas_verifier.sh

Each task runs 240 episodes (24 variations × 10 trials). Per-episode .mp4 videos land under logs/verifier_eval/eval_<task>/videos/.

Evaluation Tasks

Task	Prompt
widowx_put_eggplant_in_basket	put eggplant into yellow basket
widowx_stack_cube	stack the green cube on top of the yellow cube
widowx_carrot_on_plate	put carrot on plate
widowx_spoon_on_towel	put the spoon on the towel

How it works

1. Plan. At episode init, Gemini emits pixel-space waypoints in the policy camera's image: (u, v) plus per-waypoint tol_px and min_hold.
2. Sample. Each chunk decision draws 5 candidate action chunks from the policy (one batched forward pass through PaliGemma + action expert × 10 flow-matching steps).
3. Score. Each candidate's predicted EE endpoint is projected to pixels and scored against the upcoming K waypoints.
4. Step. Execute the best-scoring chunk, advance the active waypoint via hysteresis dwell, transit-skip, or stalled-skip.

More details

This codebase has been tested on CUDA 13 with NVIDIA RTX PRO 6000 Blackwell and RTX 5090.

Acknowledgements

The Pi0 model and PaliGemma setup are adapted from Allen's repo open-pi-zero and pytorch-paligemma. Simulation environments are from SimplerEnv. Dataset loading is from Octo and dlimp. Object detection is powered by Grounded-SAM-2.