Awesome RL-VLA for Robotic Manipulation 🤖

A curated list of papers and resources on Reinforcement Learning of Vision-Language-Action (RL-VLA) models for Robotic Manipulation. This repository provides a comprehensive overview of training paradigms, methodologies, and state-of-the-art approaches in RL-VLA research.

📢 Latest News

🔥 [November 2025] Our comprehensive survey paper "A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation" is currently under review on arXiv and will be published soon! Stay tuned for the official release.

📖 Table of Contents

• Awesome RL-VLA for Robotic Manipulation 🤖
  • 📢 Latest News
  • 📖 Table of Contents
  • 🔍 Overview
  • 🚀 Training Paradigms
    • Offline RL-VLA
    • Online RL-VLA
    • Test-time RL-VLA
  • 📚 Paper Collection
    • Legend
    • 🎯 RL-VLA Action Optimization
    • Offline RL-VLA
    • Online RL-VLA
    • Offline + Online RL-VLA
    • Test-time RL-VLA
  • 🔗 Useful Resources
    • Base VLA Models
    • Datasets & Benchmarks
    • Frameworks & Tools
  • 🤝 Contributing
    • Contribution Guidelines
  • 📄 Citation

🔍 Overview

RL training is crucial for enabling VLAs to generalize out-of-distribution (OOD) from large-scale pre-trained data. Existing RL-VLA training paradigms can be categorized into three types based on how agents obtain and utilize feedback from the environment:

• Online RL-VLA: Direct interaction with the environment during training
• Offline RL-VLA: Learning from static datasets without further environmental interaction
• Test-time RL-VLA: Models adapt their behavior during deployment without altering parameters

🚀 Training Paradigms

Offline RL-VLA

Offline RL trains VLA models on pre-collected static datasets, enabling learning independently from environment interactions. This paradigm is suitable for high-risk or resource-constrained deployment scenarios.

Key Research Directions:

• Data Utilization: Effective utilization of static datasets for policy improvement
• Objective Modification: Customizing RL objectives for novel architectures and data augmentation

Online RL-VLA

Online RL-VLA enables interactive policy learning through continuous environment interaction, empowering pre-trained VLAs with adaptive closed-loop control capability for real-world OOD environments.

Key Research Directions:

• Policy Optimization: Direct policy improvement based on environmental rewards
• Sample Efficiency: Learning effective policies with limited interaction budget
• Active Exploration: Efficient exploration strategies for higher performance gains
• Training Stability: Ensuring consistent policy updates and convergence
• Infrastructure: Scalable frameworks for online RL-VLA training

Test-time RL-VLA

Test-time RL-VLA adapts behavior during deployment through lightweight updates, addressing the expensive cost of full model fine-tuning in real-world scenarios.

Key Adaptation Mechanisms:

• Value Guidance: Using pre-trained value functions to influence action selection
• Memory Buffer Guidance: Retrieving relevant historical experiences during inference
• Planning-guided Adaptation: Explicit reasoning over future action sequences

📚 Paper Collection

Legend

• Action: AR (Autoregressive), Diffusion, Flow (Flow-matching)
• Reward: D (Dense Reward), S (Sparse Reward)
• Type: MB (Model-based), MF (Model-free)
• Environment: Sim. (Simulation), Real (Real-world)

🎯 RL-VLA Action Optimization

Different VLA architectures require distinct RL optimization strategies based on their action generation mechanisms:

• 🔤 Autoregressive VLA: Optimizes actions at the token-level. Each action token is individually optimized through RL, enabling fine-grained control over action sequences but requiring careful handling of sequential dependencies.

• 🌊 Generative VLA (Diffusion/Flow): Optimizes along the action generation process at the sequence-level. The entire action trajectory is optimized as a cohesive unit through the denoising or flow-matching process, providing holistic action optimization.

• 🔗 Dual-system VLA: Optimizes at the bridge-level. RL decides which high-level action proposal to pass to the fast controller, creating a hierarchical optimization approach that complements both token-level and sequence-level methods.

Offline RL-VLA

Method	Date	Sim.	Real	Base VLA Model	Action	Reward	Algorithm	Type	Project
Q-Transformer	2023.10	✓	✗	Transformer	AR	S	CQL	MF	🔗
PAC	2024.02	✓	✓	Perceiver-Actor-Critic	AR	S	AC	MF	🔗
ReinboT	2025.05	✓	✓	ReinboT	AR	D	DT + RTG	MF	-
CO-RFT	2025.08	✗	✓	RoboVLMs	AR	D	Cal-QL + TD3	MF	-
ARFM	2025.09	✓	✓	π₀	Flow	D	ARFM	MF	-
$π^*_{0.6}$	2025.11	✗	✓	$π_{0.6}$	Flow	D	RECAP	MF	🔗
NORA-1.5	2025.11	✓	✓	NORA-1.5	AR / Flow	D	DPO	MB	🔗

Online RL-VLA

Method	Date	Sim.	Real	Base VLA Model	Action	Reward	Algorithm	Type	Project
FLaRe	2024.09	✓	✓	SPOC	AR	S	PPO	MF	🔗
PA-RL	2024.12	✓	✓	OpenVLA	AR	S	PA-RL	MF	🔗
RLDG	2024.12	✗	✓	OpenVLA / Octo	AR / Diffusion	S	RLPD	MF	🔗
iRe-VLA	2025.01	✓	✓	iRe-VLA	AR	S	SACfD + SFT	MF	-
GRAPE	2025.02	✓	✓	OpenVLA	AR	D	TPO	MF	🔗
SafeVLA	2025.03	✓	✗	SPOC	AR	S	PPO	MF	🔗
RIPT-VLA	2025.05	✓	✗	QueST / OpenVLA-OFT	AR	S	LOOP	MF	🔗
VLA-RL	2025.05	✓	✗	OpenVLA	AR	D	PPO	MF	🔗
RLVLA	2025.05	✓	✗	OpenVLA	AR	S	PPO / GRPO / DPO	MF	🔗
RFTF	2025.05	✓	✗	GR-MG, Seer	AR	D	PPO	MF	-
TGRPO	2025.06	✓	✗	OpenVLA	AR	D	GRPO	MF	-
RLRC	2025.06	✓	✗	OpenVLA	AR	S	PPO	MF	🔗
SimpleVLA-RL	2025.09	✓	✓	OpenVLA-OFT	AR	S	GRPO	MF	🔗
Dual-Actor FT	2025.09	✓	✓	Octo / SmolVLA	Diffusion	S	QL + BC	MF	🔗
Generalist	2025.09	✓	✓	PaLI 3B	AR	D	REINFORCE	MF	🔗
VLAC	2025.09	✗	✓	VLAC	AR	D	PPO	MF	🔗
AC PPO	2025.09	✓	✗	Octo-small	AR	S	PPO+BC	MF	-
VLA-RFT	2025.10	✓	✗	VLA-Adapter	Flow	D	GRPO	MB	🔗
RLinf-VLA	2025.10	✓	✓	OpenVLA / OpenVLA-OFT	AR	S	PPO / GRPO	MF	🔗
FPO	2025.10	✓	✗	π₀	Flow	S	FPO	MF	-
ReSA	2025.10	✓	✗	OpenVLA	AR	D	PPO + SFT	MF	-
π_RL	2025.10	✓	✗	π₀ / π₀.₅	Flow	S	PPO / GRPO	MF	🔗
PLD	2025.10	✓	✓	OpenVLA / π₀ / Octo	AR / Flow	S	Cal-QL + SAC	MF	🔗
DeepThinkVLA	2025.10	✓	✗	π₀-Fast	AR	S	GRPO	MF	🔗
World-Env	2025.11	✓	✓	OpenVLA-OFT	AR	D	PPO	MB	🔗
RobustVLA	2025.11	✓	✗	OpenVLA-OFT	AR	D	PPO	MF	-
WMPO	2025.11	✓	✓	OpenVLA-OFT	AR	S	GRPO	MB	🔗

Offline + Online RL-VLA

Method	Date	Sim.	Real	Base VLA Model	Action	Reward	Algorithm	Type	Project
ConRFT	2025.04	✗	✓	Octo-small	Diffusion	S	Cal-QL + BC	MF	🔗
SRPO	2025.11	✓	✓	OpenVLA* / π₀ / π₀-Fast	AR / Flow	D	SRPO	MF (MB-Reward but MF-RL)	🔗

Test-time RL-VLA

Method	Date	Sim.	Real	Base VLA Model	Action	Reward	Algorithm	Type	Project
V-GPS	2024.10	✓	✓	Octo / RT-1 / OpenVLA	AR	D	Cal-QL	MF	🔗
Hume	2025.06	✓	✓	Hume	Flow	S	Value Guidance	MF	🔗
VLA-Reasoner	2025.09	✓	✓	OpenVLA / SpatialVLA et al.	AR / Diffusion	D	MCTS	MB	-
VLAPS	2025.11	✗	✓	Octo	AR	S	MCTS	MB	🔗

Note: The 🔗 symbol in the Project column indicates papers with available project pages, GitHub repositories, or demo websites.

🔗 Useful Resources

Base VLA Models

• OpenVLA - Open-source VLA model
• Octo - Generalist robot policy
• RT-1 - Robotics Transformer

Datasets & Benchmarks

• Open X-Embodiment - Large-scale robotic datasets
• LIBERO - Benchmark for lifelong robot learning

Frameworks & Tools

• RLinf - Infrastructure for online RL fine-tuning of VLAs

🤝 Contributing

We welcome contributions to this awesome list! Please feel free to:

1. Add new papers: Submit a PR with new RL-VLA papers following the existing format
2. Update information: Correct any errors or update paper information
3. Suggest improvements: Propose better organization or additional sections

Contribution Guidelines

• Ensure papers are relevant to RL-VLA research
• Include paper links, project pages (if available), and key details
• Follow the existing table format for consistency
• Add a brief description for new paradigms or significant methodological contributions

📄 Citation

If you find this repository useful, please consider citing:

@article{deng2025rlvla,
  title={A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation},
  author={Haoyuan Deng, Zhenyu Wu, Haichao Liu, Wenkai Guo, Yuquan Xue, Ziyu Shan, Chuanrui Zhang, Bofang Jia, Yuan Ling, Guanxing Lu, and Ziwei Wang},
  journal={arXiv preprint arXiv:},
  year={2025}
}

---

⭐ Star this repository if you find it helpful!

🔄 Watch for updates on the latest RL-VLA research!