Skip to content

Commit ad10bfd

Browse files
KraHsuKraHsu
authored
Merge pull request #11 from KraHsu/feat/inverted_pendulum
Add inverted-pendulum PPO example extension
2 parents a1efa65 + 425e718 commit ad10bfd

25 files changed

Lines changed: 1192 additions & 7 deletions

‎docs/examples/inverted-pendulum.en.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Inverted Pendulum
2+
3+
`examples/inverted_pendulum/` ships two PPO tasks for the classical cart-pole problem on flat
4+
ground. The extension mirrors `examples/unitree/`: `ManagerBasedRlEnv` over Genesis, rsl_rl PPO,
5+
a `BodyVelocitySensor` on the pole link, and the unified `genelab train` / `genelab play` CLI.
6+
7+
## Tasks
8+
9+
| Task id | Problem |
10+
|---------|---------|
11+
| `GeneLab-Inverted-Pendulum-v0` | Single inverted pole on a cart. |
12+
| `GeneLab-Double-Inverted-Pendulum-v0` | Two stacked inverted poles on a cart. |
13+
14+
## Installation
15+
16+
The extension depends on the `rl` extra (rsl_rl). Pick the `torch-*` extra that matches the
17+
hardware.
18+
19+
```bash
20+
uv sync --extra rl --extra torch-cu128
21+
uv pip install -e examples/inverted_pendulum
22+
23+
uv run genelab list tasks
24+
# -> GeneLab-Inverted-Pendulum-v0
25+
# -> GeneLab-Double-Inverted-Pendulum-v0
26+
```
27+
28+
## Single inverted pendulum
29+
30+
```bash
31+
uv run genelab train GeneLab-Inverted-Pendulum-v0 \
32+
--num-envs 4096 --max-iterations 150
33+
34+
uv run genelab play GeneLab-Inverted-Pendulum-v0 \
35+
--checkpoint logs/rsl_rl/inverted_pendulum_flat/<run>/model_150.pt --vis
36+
```
37+
38+
`--checkpoint` makes `play` route through the RL runner with `--agent trained` by default.
39+
40+
## Double inverted pendulum
41+
42+
```bash
43+
uv run genelab train GeneLab-Double-Inverted-Pendulum-v0 \
44+
--num-envs 4096 --max-iterations 300
45+
46+
uv run genelab play GeneLab-Double-Inverted-Pendulum-v0 \
47+
--checkpoint logs/rsl_rl/double_inverted_pendulum_flat/<run>/model_300.pt --vis
48+
```
49+
50+
## Sensor and underactuation
51+
52+
Only the cart slide joint is PD-controlled. The pole hinges default to `kp=0, kv=0` so the
53+
pendulum stays underactuated and the policy must learn balance through cart motion alone. A
54+
`BodyVelocitySensor` attached to the top pole supplies a noisy angular-velocity observation
55+
(corrupted with `Unoise` in the policy group, clean in the critic group).
56+
57+
## Interactive disturbance
58+
59+
Play mode launches a single environment (`num_envs=1`) and enables Genesis'
60+
`MouseInteractionPlugin`. Left-click on the cart or pole and drag — a spring force pulls the
61+
clicked link toward the cursor while the policy keeps balancing. Scroll wheel rotates the drag
62+
plane around the surface normal. Release the button to remove the force.
63+
64+
!!! tip "Smoke-test budget"
65+
A 5–10 iteration run with `--num-envs 64 --max-iterations 5` is enough to validate wiring
66+
end-to-end. The reward signal will still be noisy at that scale; convergence requires the
67+
150 / 300 iteration budgets above.
68+
69+
## Logs
70+
71+
Both tasks write to `logs/rsl_rl/<experiment>/<timestamp>_/` like the Unitree examples:
72+
73+
- `params/env.json` and `params/agent.json` — frozen configs at run time.
74+
- `model_<iter>.pt` — checkpoints saved every `save_interval` iterations.
75+
- TensorBoard event files alongside the checkpoints.
76+
77+
## See also
78+
79+
- [Unitree G1 quickstart](../getting-started/quickstart.md#unitree-g1)
80+
- [Sensors](../concepts/sensors.md)
81+
- [Play and Train CLI](../cli/play-train.md)

‎docs/examples/inverted-pendulum.zh.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# 倒立摆
2+
3+
`examples/inverted_pendulum/` 提供两个在平面上的 cart-pole 经典控制 PPO 任务。整条训练栈与
4+
`examples/unitree/` 对齐：基于 Genesis 的 `ManagerBasedRlEnv`、rsl_rl PPO、挂在杆上的
5+
`BodyVelocitySensor`，以及统一的 `genelab train` / `genelab play` CLI。
6+
7+
## 任务列表
8+
9+
| Task id | 问题 |
10+
|---------|------|
11+
| `GeneLab-Inverted-Pendulum-v0` | 小车 + 单杆的倒立摆。 |
12+
| `GeneLab-Double-Inverted-Pendulum-v0` | 小车 + 串联双杆的倒立摆。 |
13+
14+
## 安装
15+
16+
扩展依赖 `rl` extra（rsl_rl）。`torch-*` extra 按硬件挑选。
17+
18+
```bash
19+
uv sync --extra rl --extra torch-cu128
20+
uv pip install -e examples/inverted_pendulum
21+
22+
uv run genelab list tasks
23+
# -> GeneLab-Inverted-Pendulum-v0
24+
# -> GeneLab-Double-Inverted-Pendulum-v0
25+
```
26+
27+
## 单倒立摆
28+
29+
```bash
30+
uv run genelab train GeneLab-Inverted-Pendulum-v0 \
31+
--num-envs 4096 --max-iterations 150
32+
33+
uv run genelab play GeneLab-Inverted-Pendulum-v0 \
34+
--checkpoint logs/rsl_rl/inverted_pendulum_flat/<run>/model_150.pt --vis
35+
```
36+
37+
传入 `--checkpoint` 会让 `play` 自动经过 RL runner，并默认使用 `--agent trained`。
38+
39+
## 双倒立摆
40+
41+
```bash
42+
uv run genelab train GeneLab-Double-Inverted-Pendulum-v0 \
43+
--num-envs 4096 --max-iterations 300
44+
45+
uv run genelab play GeneLab-Double-Inverted-Pendulum-v0 \
46+
--checkpoint logs/rsl_rl/double_inverted_pendulum_flat/<run>/model_300.pt --vis
47+
```
48+
49+
## 传感器与欠驱动
50+
51+
只有小车的 slide 关节通过 PD 控制。两个 pole hinge 默认 `kp=0, kv=0`，保证整体处于欠驱动状态，
52+
策略必须通过小车水平运动间接稳定杆。顶端 pole 上挂载的 `BodyVelocitySensor` 给出一路带噪声的
53+
角速度观测：policy 观测组用 `Unoise` 做 corruption，critic 观测组直接读取干净值。
54+
55+
## 交互式扰动
56+
57+
`play` 默认只开 1 个环境（`num_envs=1`），并启用 Genesis 的 `MouseInteractionPlugin`。
58+
左键点击 cart 或 pole 并拖动，会有一根弹簧把所点击的 link 拉向光标位置；策略仍然在背后试图
59+
保持平衡。滚轮可绕表面法线旋转拖拽平面，松开左键即移除外力。
60+
61+
!!! tip "Smoke-test 预算"
62+
使用 `--num-envs 64 --max-iterations 5` 跑 5–10 次迭代足以验证整条链路。此时 reward 信号
63+
仍非常嘈杂，真正收敛需要上面给出的 150 / 300 次迭代预算。
64+
65+
## 日志
66+
67+
两个任务都把日志写到 `logs/rsl_rl/<experiment>/<timestamp>_/`，结构与 Unitree 示例一致：
68+
69+
- `params/env.json` 与 `params/agent.json` —— 运行时冻结的配置快照。
70+
- `model_<iter>.pt` —— 按 `save_interval` 保存的 checkpoint。
71+
- 同目录下的 TensorBoard 事件文件。
72+
73+
## See also
74+
75+
- [Unitree G1 快速开始](../getting-started/quickstart.md#unitree-g1)
76+
- [传感器](../concepts/sensors.md)
77+
- [play 与 train CLI](../cli/play-train.md)

‎docs/examples/overview.en.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,16 @@
33
The repository ships several reference extensions under `examples/`. They double as integration
44
tests for the CLI and registry.
55

6+
## inverted_pendulum
7+
8+
Two PPO cart-pole tasks built on the same `ManagerBasedRlEnv` + rsl_rl stack as the Unitree
9+
example, sized to fit in a laptop training budget:
10+
11+
- **`GeneLab-Inverted-Pendulum-v0`** — single inverted pole on a cart.
12+
- **`GeneLab-Double-Inverted-Pendulum-v0`** — two stacked inverted poles on a cart.
13+
14+
Source at `examples/inverted_pendulum/`; walkthrough at [Inverted Pendulum](inverted-pendulum.md).
15+
616
## genelab_examples
717

818
The canonical in-tree extension, wiring two tasks:

‎docs/examples/overview.zh.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,16 @@
22

33
仓库在 `examples/` 下提供数个参考扩展，同时也是 CLI 与注册表的集成测试。
44

5+
## inverted_pendulum
6+
7+
两个 PPO cart-pole 任务，训练栈与 Unitree 示例相同（`ManagerBasedRlEnv` + rsl_rl），训练预算
8+
控制在单机能跑完的量级：
9+
10+
- **`GeneLab-Inverted-Pendulum-v0`** —— 小车 + 单杆倒立摆。
11+
- **`GeneLab-Double-Inverted-Pendulum-v0`** —— 小车 + 串联双杆倒立摆。
12+
13+
源码位于 `examples/inverted_pendulum/`；完整流程见 [倒立摆](inverted-pendulum.md)。
14+
515
## genelab_examples
616

717
仓库内的标准扩展，接通两个任务：

‎examples/README.md‎

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,26 +5,36 @@ ship built-in tasks; example tasks are loaded like any other external project.
55

66
## Available Examples
77

8+
- [Inverted Pendulum](inverted_pendulum/README.md): trainable single- and double-inverted-pendulum tasks that initialize Genesis while fully exercising `train` + `play`.
89
- [GeneLab Example Extension](genelab_examples/README.md): one Python project that registers the
910
Rubik's cube and Wuji hand tasks.
1011
- [External Project](external_project/README.md): minimal standalone Python package that extends
1112
GeneLab without editing `src/genelab/`.
1213

13-
The example extension registers these task IDs:
14+
The bundled examples register these task IDs:
1415

16+
- `GeneLab-Inverted-Pendulum-v0`
17+
- `GeneLab-Double-Inverted-Pendulum-v0`
1518
- `GeneLab-Rubiks-Play-v0`
1619
- `GeneLab-Wuji-Hand-Playback-v0`
1720

18-
List example tasks from the repository root without installing the example package:
21+
List inverted-pendulum tasks from the repository root without installing the package:
22+
23+
```bash
24+
PYTHONPATH=examples/inverted_pendulum/src uv run genelab --import genelab_inverted_pendulum.tasks list tasks
25+
```
26+
27+
List the Genesis demo tasks the same way:
1928

2029
```bash
2130
PYTHONPATH=examples/genelab_examples/src uv run genelab --import genelab_examples.tasks list tasks
2231
```
2332

24-
Install the example extension once if you want `uv run genelab list tasks` to load it through entry
33+
Install an example extension once if you want `uv run genelab list tasks` to load it through entry
2534
points:
2635

2736
```bash
37+
uv pip install -e examples/inverted_pendulum
2838
uv pip install -e examples/genelab_examples
2939
uv run genelab list tasks
3040
```
@@ -43,21 +53,31 @@ override keys are converted to underscores.
4353
Examples:
4454

4555
```bash
56+
PYTHONPATH=examples/inverted_pendulum/src uv run genelab --import genelab_inverted_pendulum.tasks train GeneLab-Inverted-Pendulum-v0 --num-envs 4096 --max-iterations 150
57+
PYTHONPATH=examples/inverted_pendulum/src uv run genelab --import genelab_inverted_pendulum.tasks play GeneLab-Inverted-Pendulum-v0 --checkpoint logs/rsl_rl/inverted_pendulum_flat/<run>/model_150.pt --vis
4658
PYTHONPATH=examples/genelab_examples/src uv run genelab --import genelab_examples.tasks play GeneLab-Rubiks-Play-v0 --steps 5 --env.robot.cubie_size 0.04 --env.robot.gap 0.002
4759
PYTHONPATH=examples/genelab_examples/src uv run genelab --import genelab_examples.tasks play GeneLab-Rubiks-Play-v0 --env.robot.welded true
4860
PYTHONPATH=examples/genelab_examples/src uv run genelab --import genelab_examples.tasks play GeneLab-Wuji-Hand-Playback-v0 --env.reset_interval 0
4961
```
5062

51-
`train` validates the task id and configuration path but currently reports that training is not
52-
implemented:
63+
`train` is implemented for the inverted-pendulum tasks. The Rubik's cube and Wuji hand demo tasks are
64+
play-only and report that training is not implemented:
5365

5466
```bash
5567
PYTHONPATH=examples/genelab_examples/src uv run genelab --import genelab_examples.tasks train GeneLab-Rubiks-Play-v0
5668
```
5769

5870
## Smoke Tests
5971

60-
Run short headless smoke tests after the Genesis assets and cache have initialized:
72+
Run the inverted-pendulum smoke tests first; a tiny rsl_rl run exercises the full Genesis +
73+
PPO pipeline:
74+
75+
```bash
76+
PYTHONPATH=examples/inverted_pendulum/src uv run genelab --import genelab_inverted_pendulum.tasks train GeneLab-Inverted-Pendulum-v0 --num-envs 64 --max-iterations 5
77+
PYTHONPATH=examples/inverted_pendulum/src uv run genelab --import genelab_inverted_pendulum.tasks train GeneLab-Double-Inverted-Pendulum-v0 --num-envs 64 --max-iterations 5
78+
```
79+
80+
Then run short headless smoke tests after the Genesis assets and cache have initialized:
6181

6282
```bash
6383
PYTHONPATH=examples/genelab_examples/src uv run genelab --import genelab_examples.tasks play GeneLab-Rubiks-Play-v0 --steps 5

‎examples/inverted_pendulum/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Inverted pendulum examples
2+
3+
GeneLab extension that ships two PPO tasks for the classical cart-pole problem on flat ground:
4+
5+
- **`GeneLab-Inverted-Pendulum-v0`** — balance a single inverted pole on a cart.
6+
- **`GeneLab-Double-Inverted-Pendulum-v0`** — stabilise two stacked inverted poles on a cart.
7+
8+
The training stack mirrors `examples/unitree/`: `ManagerBasedRlEnv` over Genesis, rsl_rl PPO,
9+
`BodyVelocitySensor` for the noisy pole rate observation, and the unified `genelab train` /
10+
`genelab play` CLI.
11+
12+
## Layout
13+
14+
```
15+
examples/inverted_pendulum/
16+
├── pyproject.toml
17+
├── README.md
18+
├── assets/ # cart-pole MJCFs
19+
│ ├── inverted_pendulum.xml
20+
│ └── double_inverted_pendulum.xml
21+
└── src/genelab_inverted_pendulum/
22+
├── tasks.py # registers both tasks
23+
├── mdp.py # cart-pole-specific reward / termination / event terms
24+
├── single/ # single-pendulum config (robot + env + PPO)
25+
└── double/ # double-pendulum config (robot + env + PPO)
26+
```
27+
28+
## Quickstart
29+
30+
```bash
31+
# From the GeneLab repo root
32+
uv sync --extra rl --extra torch-cu128 # pick whichever torch flavor fits your GPU
33+
uv pip install -e examples/inverted_pendulum
34+
35+
uv run genelab list tasks
36+
# -> GeneLab-Inverted-Pendulum-v0
37+
# -> GeneLab-Double-Inverted-Pendulum-v0
38+
```
39+
40+
### Single inverted pendulum
41+
42+
```bash
43+
uv run genelab train GeneLab-Inverted-Pendulum-v0 --num_envs 4096 --max_iterations 150
44+
uv run genelab play GeneLab-Inverted-Pendulum-v0 \
45+
--checkpoint logs/rsl_rl/inverted_pendulum_flat/<run>/model_150.pt
46+
```
47+
48+
### Double inverted pendulum
49+
50+
```bash
51+
uv run genelab train GeneLab-Double-Inverted-Pendulum-v0 --num_envs 4096 --max_iterations 300
52+
uv run genelab play GeneLab-Double-Inverted-Pendulum-v0 \
53+
--checkpoint logs/rsl_rl/double_inverted_pendulum_flat/<run>/model_300.pt
54+
```
55+
56+
## Notes
57+
58+
- Only the cart slide joint is PD-controlled. The pole hinges default to `kp=0, kv=0` so the
59+
pendulum stays underactuated and the policy must learn balance via cart motion alone.
60+
- The observation group corrupts joint position, joint velocity, and pole angular velocity with
61+
`Unoise`. The critic group sees the same features without corruption.
62+
- Logs land under `logs/rsl_rl/<experiment>/<timestamp>_/` exactly like the Unitree examples,
63+
with `params/env.json`, `params/agent.json`, and `model_<iter>.pt` files.

‎examples/inverted_pendulum/assets/double_inverted_pendulum.xml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
<mujoco model="genelab_double_inverted_pendulum">
2+
<option timestep="0.005" gravity="0 0 -9.81" integrator="implicitfast"/>
3+
<default>
4+
<joint armature="0" damping="0"/>
5+
<geom contype="0" conaffinity="0"/>
6+
</default>
7+
<worldbody>
8+
<body name="cart" pos="0 0 0.12">
9+
<joint name="cart_slide" type="slide" axis="1 0 0" range="-3.0 3.0" damping="0.05"/>
10+
<!-- contype/conaffinity bits picked so each geom is present in the rigid solver
11+
(visible to the mouse-interaction raycaster) but never collides with the
12+
plane or with sibling links. -->
13+
<geom name="cart_box" type="box" size="0.17 0.11 0.05" mass="1.0"
14+
rgba="0.10 0.45 0.85 1" contype="2" conaffinity="2"/>
15+
<body name="pole_1" pos="0 0 0.05">
16+
<joint name="pole_1_hinge" type="hinge" axis="0 1 0" range="-1.2 1.2" damping="0.002"/>
17+
<geom name="pole_1_capsule" type="capsule" fromto="0 0 0 0 0 0.55" size="0.023"
18+
mass="0.1" rgba="0.95 0.35 0.15 1" contype="4" conaffinity="4"/>
19+
<body name="pole_2" pos="0 0 0.55">
20+
<joint name="pole_2_hinge" type="hinge" axis="0 1 0" range="-1.5 1.5" damping="0.002"/>
21+
<geom name="pole_2_capsule" type="capsule" fromto="0 0 0 0 0 0.45" size="0.020"
22+
mass="0.08" rgba="0.95 0.65 0.12 1" contype="8" conaffinity="8"/>
23+
</body>
24+
</body>
25+
</body>
26+
</worldbody>
27+
<actuator>
28+
<motor name="cart_force" joint="cart_slide" gear="1"/>
29+
</actuator>
30+
</mujoco>

‎examples/inverted_pendulum/assets/inverted_pendulum.xml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
<mujoco model="genelab_inverted_pendulum">
2+
<option timestep="0.005" gravity="0 0 -9.81" integrator="implicitfast"/>
3+
<default>
4+
<joint armature="0" damping="0"/>
5+
<geom contype="0" conaffinity="0"/>
6+
</default>
7+
<worldbody>
8+
<body name="cart" pos="0 0 0.12">
9+
<joint name="cart_slide" type="slide" axis="1 0 0" range="-3.0 3.0" damping="0.05"/>
10+
<!-- contype/conaffinity bits picked so the geom is present in the rigid solver
11+
(visible to the mouse-interaction raycaster) but never participates in a
12+
collision pair with the plane or the pole. -->
13+
<geom name="cart_box" type="box" size="0.15 0.10 0.05" mass="1.0"
14+
rgba="0.10 0.45 0.85 1" contype="2" conaffinity="2"/>
15+
<body name="pole" pos="0 0 0.05">
16+
<joint name="pole_hinge" type="hinge" axis="0 1 0" range="-1.2 1.2" damping="0.002"/>
17+
<geom name="pole_capsule" type="capsule" fromto="0 0 0 0 0 0.80" size="0.025"
18+
mass="0.1" rgba="0.95 0.35 0.15 1" contype="4" conaffinity="4"/>
19+
</body>
20+
</body>
21+
</worldbody>
22+
<actuator>
23+
<motor name="cart_force" joint="cart_slide" gear="1"/>
24+
</actuator>
25+
</mujoco>

0 commit comments

Comments
 (0)