Pi0 and Lerobot with Rerun

A repo to explore training robots with Pi0 and Lerobot and human pose motion retargeting

Installation

Currently only linux is supported

Using Pixi

Make sure you have the Pixi package manager installed

git clone https://github.com/rerun-io/pi0-lerobot.git
cd pi0-lerobot

Usage

Human Pose and Kinematics

For the full assembly101 dataset go to this link to get/download dataset

To run dataset visualization

pixi run visualize-assembly101-dataset

To run 2D pose estimation, tracking, and triangulation

pixi run pose-estimation-assembly101

Jupyter Notebook Tutorials

pixi run notebook_tutorial

TODO

Part 1. Human Pose and Kinematics

• Basic Triangulation from 2D Detection (body pose)
• Basic Triangulation from 2D detection (hand pose)
• Detection by Tracking (extrapolate 3d views-> Generate bounding box based on extrapolated -> check based on kpts confidence)
• Mano + SMPL fitting for skeleton kinematics
• Add EgoCentric (first person) headset views and poses

Part 2. Body and Hand Pose Retargeting

• Implement pose retargeting to isaac sim

Part 3. Immitation Learning and Teleoperation

• Notebook Tutorials
• Finetune and evaluation on Aloha simulator with Pi0
• Finetune base Pi0 on SO-ARM100 from lerobot