skyrl-train

SkyRL-Train: A modular, performant RL framework for post-training LLMs

Important

Note: SkyRL is undergoing a repo reorganization into the SkyRL/skyrl folder, which unifies the skyrl libraries (skyrl-train, skyrl-tx) into a single package. The code that was previously in the skyrl-train package can now be found in skyrl/{backends/, train/, utils/}. See issue: #1145

Overview

With a focus on modularity, skyrl-train makes it easy to prototype new training algorithms, environments, and execution plans—without compromising usability or speed.

skyrl-train is for users who want to modify anything:

• Quickly develop new environments without modifying or understanding the training code.
• Modify the training execution plan such as model placement, colocation or disaggregation of training and generation, and async RL.
• Implement custom trajectory generation specific to your use-case, such as custom sampling methods, tree search, etc.
• … make any other flexible modifications to the RL workflow!

Key Features

The skyrl-train package supports:

• PPO and GRPO
• Training Backends: FSDP, FSDP2, and Megatron
• Inference backends: vLLM, SGLang, and any custom OpenAI API compatible endpoint that exposes a method to perform weight sync
• Ulysses sequence parallelism for long-context training
• Colocated or disaggregated training and generation (including on heterogeneous hardware)
• Synchronous RL, async one-off pipelining, or fully async RL with in-flight weight updates
• Simple batched rollouts or Asynchronous rollouts for multi-turn conversations
• Weight sync via NCCL, gloo, or checkpoint-and-load
• Integration with skyrl-gym, verifiers, OpenEnv, Harbor/Terminal-Bench, and more!
• Sequence packing and Flash Attention 2
• Algorithmic support for RLOO, REINFORCE, GSPO, CISPO, SAPO
• Step-wise training for fully on policy multi-turn RL
• 5D Parallelism support for MoE models with the Megatron backend

Documentation

Find documentation at: docs.skyrl.ai/docs/

Quick Start

A quick start guide for installation and your first training run is provided below.

Requirements

The only requirements are:

• CUDA version 12.8
• uv

If you're running on an existing Ray cluster, make sure to use Ray 2.51.1 and Python 3.12. If not, proceed with the installation instructions below.

First, clone the repository:

git clone --recurse-submodules https://github.com/NovaSky-AI/SkyRL
cd SkyRL/

Then, create a new virtual environment and install the dependencies:

# creates a venv at .venv/
uv sync --extra fsdp 
source .venv/bin/activate

Then, prepare the dataset:

uv run -- python examples/train/gsm8k/gsm8k_dataset.py

Finally, before training, make sure to configure Ray to use uv:

export RAY_RUNTIME_ENV_HOOK=ray._private.runtime_env.uv_runtime_env_hook.hook
# or add to your .bashrc
# echo 'export RAY_RUNTIME_ENV_HOOK=ray._private.runtime_env.uv_runtime_env_hook.hook' >> ~/.bashrc

You should now be able to run our example script (assumes at least 4 GPUs):

export WANDB_API_KEY=<your wandb api key>
bash examples/train/gsm8k/run_gsm8k.sh

For detailed installation instructions, as well as more examples, please refer to our documentation.

Training on a new task or environment

To implement a new task or environment using the SkyRL-Gym interface, please see our Walkthrough Docs.

If you don't want to use the SkyRL-Gym interface, or you have an existing task or agentic pipeline implementation and just want to train with it on top of SkyRL, we recommend you create a simple custom Generator, which requires implementing a single method, generate(). We have one example of a custom Generator at SkyRLGymGenerator which executes environments written in the SkyRL-Gym interface. We are working to provide more example integrations of agent harnesses -- please reach out if you'd like yours to be one of them!

Reproducing SkyRL-SQL

We also test SkyRL by reproducing our prior release SkyRL-SQL, which enabled efficient Multi-Turn RL for Text2SQL. You can find a link to the wandb report here, and a detailed walk through of the reproduction in our documentation.

Acknowledgement

This work is done at Berkeley Sky Computing Lab in collaboration with Anyscale, with generous compute support from Anyscale, Databricks, NVIDIA, Lambda Labs, and AMD.

We adopt many lessons and code from several great projects such as veRL, OpenRLHF, Search-R1, OpenReasonerZero, and NeMo-RL. We appreciate each of these teams and their contributions to open-source research!

Citation

If you find the work in skyrl-train helpful, please consider citing:

@misc{griggs2025skrylv01,
      title={Evolving SkyRL into a Highly-Modular RL Framework},
      author={Tyler Griggs and Sumanth Hegde and Eric Tang and Shu Liu and Shiyi Cao and Dacheng Li and Charlie Ruan and Philipp Moritz and Kourosh Hakhamaneshi and Richard Liaw and Akshay Malik and Matei Zaharia and Joseph E. Gonzalez and Ion Stoica},
      year={2025},
      note={Notion Blog}
}