Isaac Lab Evaluation Tasks

📝 Overview

This repository introduces two new industrial manipulation tasks designed in Isaac Lab, enabling simulating and evaluating manipulation policies (e.g. Isaac GR00T N1) using a humanoid robot. The tasks are designed to simulate realistic industrial scenarios, including Nut Pouring and Exhaust Pipe Sorting. It also provides benchmarking scripts for closed-loop evaluation of manipulation policy (i.e. Isaac GR00T N1) with post-trained checkpoints. These scripts enable developers to load prebuilt Isaac Lab environments and industrial tasks—such as nut pouring and pipe sorting—and run standardized benchmarks to quantitatively assess policy performance.

📦 Installation

For detailed installation instructions, see Installation Guide.

🛠️ Evaluation Tasks

Two industrial tasks have been created in Isaac Lab to simulate robotic manipulation scenarios. The environments are set up with a humanoid robot (i.e. Fourier GR1-T2) positioned in front of several industrial objects on a table. This can include multi-step bi-manual tasks such as grasping, moving, sorting, or placing the objects into specific locations.

The robot is positioned upright, facing the table with both arms slightly bent and hands open. A first-person-view monocular RGB camera is mounted on its head to cover the workspace.

Nut Pouring

The robot picks up a beaker containing metallic nuts, pours one nut into a bowl, and places the bowl on a scale.

The task is defined as successful if following criteria have been met.

1. The sorting beaker is placed in the sorting bin
2. The factory nut is in the sorting bowl
3. The sorting bowl is placed on the sorting scale

Exhaust Pipe Sorting

The robot picks up the blue exhaust pipe, transfers it to the other hand, and places the pipe into the blue bin.

The task is defined as successful if following criteria has been met.

1. The blue exhaust pipe is placed in the correct position

📦 Downloading Datasets (Optional)

For dataset information and download instructions, see Datasets Guide.

🤖 Isaac GR00T N1 Policy Post Training (Optional)

For detailed post-training instructions including data conversion and model fine-tuning, see Post Training Guide.

📦 Downloading Checkpoints

For information on available pre-trained checkpoints and download instructions, see Checkpoints Guide.

📈 Policy Closed-loop Evaluation

For detailed evaluation instructions including benchmarking features and performance results, see Evaluation Guide.

Code formatting

We have a pre-commit template to automatically format your code. To install pre-commit:

pip install pre-commit

Then you can run pre-commit with:

pre-commit run --all-files

Troubleshooting

For common issues and solutions, see Troubleshooting Guide.

Contributing

For more details, see CONTRIBUTING.md