Skip to content

Commit 293f946

Browse files
KraHsuKraHsu
authored
Merge pull request #31 from KraHsu/docs/mdp-terms-reference
Add MDP terms reference bilingual doc
2 parents b7afcee + 359c80f commit 293f946

4 files changed

Lines changed: 561 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,11 @@ trajectory so breaking changes can land in any minor release until the 1.0 stabi
88

99
### Added
1010

11+
- `docs/concepts/mdp.{en,zh}.md`: bilingual reference for the reusable MDP term
12+
library (`genelab.mdp.{actions,commands,observations,rewards,terminations,
13+
events,curriculums,noise}`). Every re-exported function / class is tabulated
14+
with its required params, output shape, and a minimal wiring example, plus
15+
warnings on the `joint_acc_l2` / `feet_air_time` placeholder rewards.
1116
- `docs/concepts/managers.{en,zh}.md`, `docs/cli/list-info.{en,zh}.md`: bilingual
1217
manager-system concept page (seven managers + `ManagerTermBaseCfg` + term
1318
lifecycle + `EventMode` semantics + per-manager wiring examples) and CLI

‎docs/concepts/mdp.en.md‎

Lines changed: 281 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,281 @@
1+
# MDP terms reference
2+
3+
`genelab.mdp` is a library of plain Python callables that plug straight into
4+
the seven managers. Every function follows the same signature shape — `(env,
5+
**params) -> torch.Tensor` for observation / reward / termination, `(env,
6+
env_ids, **params) -> None` for events, `(env, env_ids, **params) -> Tensor`
7+
for curriculums — and is referenced by the corresponding `*TermCfg.func` (or
8+
`class_type`) field.
9+
10+
## Reusable building blocks
11+
12+
Every name listed below is re-exported by `genelab.mdp.__init__`, so
13+
`from genelab import mdp` is enough to reach them. The full module map:
14+
15+
```
16+
genelab.mdp.actions → JointPositionAction(Cfg)
17+
genelab.mdp.commands → UniformVelocityCommand(Cfg), MotionCommand(Cfg), MotionLoader
18+
genelab.mdp.observations → 16 observation functions
19+
genelab.mdp.rewards → 15 reward functions / classes
20+
genelab.mdp.terminations → 6 termination functions
21+
genelab.mdp.events → 3 event functions
22+
genelab.mdp.curriculums → 1 curriculum function
23+
genelab.mdp.noise → NoiseCfg base + Unoise + Gnoise
24+
```
25+
26+
The output-shape column below uses `B = num_envs`, `D = action / joint /
27+
sensor dimension`, `N = number of bodies or feet`. Reward and termination
28+
terms always return shape `(B,)`; observation terms return `(B, D)` (or are
29+
auto-unsqueezed from `(B,)` by `ObservationManager`).
30+
31+
## Actions
32+
33+
| Name | Signature (Cfg fields used) | Behavior |
34+
|---|---|---|
35+
| `JointPositionAction` | `JointPositionActionCfg(joint_names, scale, use_default_offset, asset_name)` | `target = default + scale * raw_action`. Targets are dispatched via `Articulation.write_joint_targets_partial`, which routes each joint to its actuator group's declared channel (position for implicit PD, force for `IdealPDActuator` / `DCMotorActuator`). `scale=None` (default) inherits per-joint scale from `Articulation.action_scale_tensor`; a `float` or `dict[str, float]` overrides it. Patterns are regex; unmatched patterns raise. |
36+
37+
```python
38+
from genelab.mdp.actions.joint_position import JointPositionActionCfg
39+
40+
actions_cfg = {
41+
"panda_arm": JointPositionActionCfg(
42+
asset_name="robot",
43+
joint_names=(r"^joint[1-7]$",),
44+
use_default_offset=True,
45+
),
46+
}
47+
```
48+
49+
## Commands
50+
51+
| Name | Signature (Cfg fields used) | Behavior |
52+
|---|---|---|
53+
| `UniformVelocityCommand` | `UniformVelocityCommandCfg(ranges, rel_standing_envs, heading_command, heading_control_stiffness, resampling_time_range, asset_name)` | Body-frame `[lin_vel_x, lin_vel_y, ang_vel_z]` per env, resampled uniformly from `ranges` every `resampling_time_range` seconds. With `heading_command=True` (default), `ang_vel_z` is rewritten each step to drive the body toward a sampled heading. `rel_standing_envs` zeroes a fraction of resampled commands so the policy also sees a "stand still" target. `command` shape `(B, 3)`. |
54+
| `MotionCommand` | `MotionCommandCfg(motion_file, anchor_body_name, body_names, pose_range, velocity_range, sampling_mode, ...)` | Drives the env toward a recorded motion clip frame-by-frame. Reads the NPZ schema produced by mjlab's `csv_to_npz`. Exposes anchor + multi-body reference and current pose / velocity for the relative-pose reward functions in the next section. `sampling_mode="start"` always begins at frame 0; `"uniform"` (default) samples a random frame. |
55+
| `MotionLoader` | `MotionLoader(motion_file, body_indexes, device)` | Helper class consumed by `MotionCommand`. Loads the NPZ into device tensors and slices `body_pos_w / body_quat_w / body_lin_vel_w / body_ang_vel_w` down to the bodies the command tracks. Direct use is rare — instantiate `MotionCommandCfg` instead. |
56+
57+
```python
58+
from genelab.mdp.commands.velocity_command import UniformVelocityCommandCfg
59+
60+
commands_cfg = {
61+
"base_velocity": UniformVelocityCommandCfg(
62+
asset_name="robot",
63+
resampling_time_range=(10.0, 10.0),
64+
ranges=UniformVelocityCommandCfg.Ranges(
65+
lin_vel_x=(-1.0, 1.0), lin_vel_y=(-0.5, 0.5), ang_vel_z=(-1.0, 1.0),
66+
),
67+
heading_command=True,
68+
),
69+
}
70+
```
71+
72+
## Observations
73+
74+
All listed functions take `env` plus the noted kwargs and return a tensor of
75+
shape `(B, D)` or `(B,)` (auto-unsqueezed to `(B, 1)` by the manager).
76+
77+
| Name | Params | Shape | Source |
78+
|---|---|---|---|
79+
| `base_lin_vel` | — | `(B, 3)` | Body-frame floating-base linear velocity. |
80+
| `base_ang_vel` | — | `(B, 3)` | Body-frame floating-base angular velocity. |
81+
| `projected_gravity` | — | `(B, 3)` | World gravity projected into the body frame (IMU-orientation proxy). |
82+
| `joint_pos_rel` | — | `(B, num_dofs)` | Joint positions minus the default pose. |
83+
| `joint_vel_rel` | — | `(B, num_dofs)` | Joint velocities (default is zero — same as raw `joint_vel`). |
84+
| `last_action` | — | `(B, total_action_dim)` | The action tensor the manager processed last step. |
85+
| `generated_commands` | `command_name` | `(B, command_dim)` | `env.command_manager.get_command(command_name)`. |
86+
| `sensor_data` | `sensor_name` | varies | Per-step cached tensor on `env.sensors[sensor_name]`. Use this for IMU, FrameTransformer, ray-cast outputs that are already tensors. |
87+
| `foot_air_time` | `sensor_name` (must be a `ContactSensor`) | `(B, N_feet)` | Current per-foot air time (zero while in contact). |
88+
| `foot_contact` | `sensor_name` (must be a `ContactSensor`) | `(B, N_feet)` | Binary contact mask as float. |
89+
| `foot_contact_forces` | `sensor_name` (must be a `ContactSensor`) | `(B, N_feet * 3)` | Per-foot contact force, compressed via `sign(f) * log1p(|f|)` and flattened. |
90+
| `height_scan` | `sensor_name` (must be a `TerrainHeightSensor`) | `(B, num_rays)` | Per-ray heights above the terrain. |
91+
| `motion_anchor_pos_b` | `command_name` (must be a `MotionCommand`) | `(B, 3)` | Reference anchor position expressed in the robot's anchor frame. |
92+
| `motion_anchor_ori_b` | `command_name` | `(B, 6)` | Reference 6D orientation (first two columns of the rotation matrix). |
93+
| `robot_body_pos_b` | `command_name` | `(B, N_bodies * 3)` | Per-body positions in the robot's anchor frame (privileged critic obs). |
94+
| `robot_body_ori_b` | `command_name` | `(B, N_bodies * 6)` | Per-body 6D orientations in the robot's anchor frame. |
95+
96+
```python
97+
from genelab.managers import ObservationGroupCfg, ObservationTermCfg
98+
from genelab import mdp
99+
from genelab.mdp.noise import Unoise
100+
101+
observations_cfg = {
102+
"policy": ObservationGroupCfg(
103+
terms={
104+
"base_lin_vel": ObservationTermCfg(func=mdp.base_lin_vel,
105+
noise=Unoise(n_min=-0.1, n_max=0.1)),
106+
"joint_pos_rel": ObservationTermCfg(func=mdp.joint_pos_rel),
107+
"last_action": ObservationTermCfg(func=mdp.last_action),
108+
"commands": ObservationTermCfg(func=mdp.generated_commands,
109+
params={"command_name": "base_velocity"}),
110+
},
111+
enable_corruption=True,
112+
),
113+
}
114+
```
115+
116+
## Rewards
117+
118+
All reward functions return shape `(B,)`. `RewardManager` multiplies by
119+
`RewardTermCfg.weight` and (when `scale_rewards_by_dt=True`) by `dt`. Negative
120+
weights turn a "deviation" reward into a penalty.
121+
122+
| Name | Params | Behavior |
123+
|---|---|---|
124+
| `track_linear_velocity_xy_exp` | `command_name`, `std=0.5` | `exp(-||cmd_xy − vel_xy||² / std²)`. |
125+
| `track_angular_velocity_z_exp` | `command_name`, `std=0.5` | `exp(-(cmd_z − vel_z)² / std²)`. |
126+
| `action_rate_l2` | — | `Σ_d (action_d − prev_action_d)²` — penalize action jitter. |
127+
| `joint_acc_l2` | — | **Placeholder** (currently returns zeros; awaits accel buffer). |
128+
| `flat_orientation_l2` | — | `Σ (projected_gravity_xy)²` — penalize tilt. |
129+
| `upright_exp` | `std=0.45` | `exp(-||projected_gravity_xy||² / std²)` — positive reward for upright base. |
130+
| `variable_posture` | `command_name`, `std_standing` / `std_walking` / `std_running` (regex→float dicts), `default_std`, `walking_threshold`, `running_threshold` | Speed-dependent posture reward: `exp(-mean((joint_pos − default)² / std²))`, with std chosen per env from the command magnitude. Class-style term — instantiated once at construct, then `__call__` per step. |
131+
| `joint_pos_limits` | — | L2 of joint-position excursion past ±π. |
132+
| `feet_air_time` | `threshold=0.4` | **Stub** — reward proportional to mean foot-link height above ground, clamped at `threshold`. |
133+
| `motion_global_anchor_position_error_exp` | `command_name`, `std` | `exp(-||p_ref − p_robot||² / std²)` on the anchor body in world frame. |
134+
| `motion_global_anchor_orientation_error_exp` | `command_name`, `std` | Geodesic rotation error on the anchor body, mapped through a Gaussian kernel. |
135+
| `motion_relative_body_position_error_exp` | `command_name`, `std`, `body_names=None` | Multi-body L2 position error against anchor-aligned reference frames. `body_names=None` includes every tracked body. |
136+
| `motion_relative_body_orientation_error_exp` | `command_name`, `std`, `body_names=None` | Multi-body geodesic rotation error against anchor-aligned references. |
137+
| `motion_global_body_linear_velocity_error_exp` | `command_name`, `std`, `body_names=None` | L2 world-frame linear-velocity error across the tracked bodies. |
138+
| `motion_global_body_angular_velocity_error_exp` | `command_name`, `std`, `body_names=None` | L2 world-frame angular-velocity error across the tracked bodies. |
139+
140+
!!! warning "Placeholder rewards"
141+
`joint_acc_l2` and `feet_air_time` are stubs awaiting proper accel /
142+
contact buffers. They compile and emit a tensor of the correct shape but
143+
the *value* is not the locomotion-grade signal one would normally expect
144+
from these names. Treat them as `weight=0.0` placeholders until they are
145+
wired against per-step velocity history (`joint_acc_l2`) and a
146+
`ContactSensor` (`feet_air_time` — `mdp.foot_air_time` already provides
147+
the real signal as an observation term).
148+
149+
```python
150+
from genelab.managers import RewardTermCfg
151+
from genelab import mdp
152+
153+
rewards_cfg = {
154+
"track_lin_vel": RewardTermCfg(
155+
func=mdp.track_linear_velocity_xy_exp,
156+
weight=1.0,
157+
params={"command_name": "base_velocity", "std": 0.25},
158+
),
159+
"track_ang_vel": RewardTermCfg(
160+
func=mdp.track_angular_velocity_z_exp,
161+
weight=0.5,
162+
params={"command_name": "base_velocity", "std": 0.25},
163+
),
164+
"upright": RewardTermCfg(func=mdp.upright_exp, weight=0.2),
165+
"action_rate": RewardTermCfg(func=mdp.action_rate_l2, weight=-0.005),
166+
"flat_ori": RewardTermCfg(func=mdp.flat_orientation_l2, weight=-0.5),
167+
}
168+
```
169+
170+
## Terminations
171+
172+
All termination functions return shape `(B,)` bool. `TerminationTermCfg.time_out`
173+
routes the value to the truncation buffer (RSL-RL's "info['time_out']") instead
174+
of the terminated buffer.
175+
176+
| Name | Params | Behavior |
177+
|---|---|---|
178+
| `time_out` | — | `episode_length_buf >= max_episode_length`. Always paired with `time_out=True`. |
179+
| `bad_orientation` | `limit_angle=math.radians(70.0)` | True when the body z-axis tilts more than `limit_angle` from world up. |
180+
| `root_height_below` | `min_height` | True when `root_pos.z < min_height`. |
181+
| `bad_anchor_pos_z_only` | `command_name`, `threshold` | True when the robot anchor z drifts further than `threshold` from the reference clip. |
182+
| `bad_anchor_ori` | `command_name`, `threshold` | True when the tilt error between robot anchor and reference exceeds `threshold` (gravity-z proxy). |
183+
| `bad_motion_body_pos_z_only` | `command_name`, `threshold`, `body_names=None` | True when any selected body's vertical position deviates past `threshold`. |
184+
185+
```python
186+
from genelab.managers import TerminationTermCfg
187+
from genelab import mdp
188+
189+
terminations_cfg = {
190+
"time_out": TerminationTermCfg(func=mdp.time_out, time_out=True),
191+
"fell_over": TerminationTermCfg(func=mdp.bad_orientation,
192+
params={"limit_angle": 1.0}),
193+
}
194+
```
195+
196+
## Events
197+
198+
Event functions take an extra `env_ids` argument and return `None`. The
199+
`EventTermCfg.mode` decides when they fire — `startup` once at construct,
200+
`reset` on every env reset, `interval` on per-env countdown (requires
201+
`interval_range_s`).
202+
203+
| Name | Params | Behavior |
204+
|---|---|---|
205+
| `reset_root_state_uniform` | `pose_range` (`x` / `y` / `z` / `roll` / `pitch` / `yaw` → `(low, high)` dict), `velocity_range` (same axes) | Randomize floating-base pose and velocity within the given ranges. Pose offsets are added to `cfg.robot.init_pos`; orientation offsets layer on top of `cfg.robot.init_quat`. |
206+
| `reset_joints_to_default` | `pos_jitter=0.0`, `vel_jitter=0.0` | Write the default joint pose to the selected envs, optionally with uniform ±jitter on position and velocity. |
207+
| `push_by_setting_velocity` | `velocity_range` (`x` / `y` / `z` / `roll` / `pitch` / `yaw` → `(low, high)`) | Overwrite the base linear and angular velocity. Combined with `mode="interval"`, this is the canonical "random push" disturbance. |
208+
209+
```python
210+
from genelab.managers import EventTermCfg
211+
from genelab import mdp
212+
213+
events_cfg = {
214+
"reset_root": EventTermCfg(
215+
mode="reset",
216+
func=mdp.reset_root_state_uniform,
217+
params={"pose_range": {"yaw": (-3.14, 3.14)},
218+
"velocity_range": {"x": (-0.1, 0.1)}},
219+
),
220+
"push_robot": EventTermCfg(
221+
mode="interval",
222+
func=mdp.push_by_setting_velocity,
223+
interval_range_s=(8.0, 12.0),
224+
params={"velocity_range": {"x": (-0.5, 0.5), "y": (-0.5, 0.5)}},
225+
),
226+
}
227+
```
228+
229+
## Curriculums
230+
231+
| Name | Params | Behavior |
232+
|---|---|---|
233+
| `terrain_levels_vel` | `distance_threshold`, `demote_ratio=0.5` | Promote / demote each env's `TerrainImporter.terrain_levels` index by how far it walked from spawn. Envs that travelled more than `distance_threshold` move up; envs that travelled less than `distance_threshold * demote_ratio` move down. Levels then drive a fresh spawn-origin lookup, and the curriculum writes the new root pose into the sim. No-op when `env.scene.terrain is None`. Returns the per-env mean level so the manager logs `Curriculum/<term-name>`. |
234+
235+
```python
236+
from genelab.managers import CurriculumTermCfg
237+
from genelab.mdp.curriculums import terrain_levels_vel
238+
239+
curriculum_cfg = {
240+
"terrain_levels": CurriculumTermCfg(
241+
func=terrain_levels_vel,
242+
params={"distance_threshold": 5.0, "demote_ratio": 0.5},
243+
),
244+
}
245+
```
246+
247+
## Noise
248+
249+
`NoiseCfg` is the abstract base; concrete subclasses plug into
250+
`ObservationTermCfg.noise` and only fire when the enclosing
251+
`ObservationGroupCfg.enable_corruption=True`.
252+
253+
| Name | Fields | Behavior |
254+
|---|---|---|
255+
| `NoiseCfg` | — | Abstract base; subclass and implement `apply(data) -> Tensor`. |
256+
| `Unoise` | `n_min=-1.0`, `n_max=1.0` | Uniform additive noise sampled in `[n_min, n_max]`. |
257+
| `Gnoise` | `mean=0.0`, `std=1.0` | Gaussian additive noise `N(mean, std²)`. |
258+
259+
```python
260+
from genelab.managers import ObservationGroupCfg, ObservationTermCfg
261+
from genelab.mdp.noise import Gnoise, Unoise
262+
from genelab import mdp
263+
264+
observations_cfg = {
265+
"policy": ObservationGroupCfg(
266+
enable_corruption=True,
267+
terms={
268+
"base_lin_vel": ObservationTermCfg(func=mdp.base_lin_vel,
269+
noise=Unoise(n_min=-0.1, n_max=0.1)),
270+
"joint_pos_rel": ObservationTermCfg(func=mdp.joint_pos_rel,
271+
noise=Gnoise(std=0.01)),
272+
},
273+
),
274+
}
275+
```
276+
277+
## See also
278+
279+
- [Managers and MDP terms](managers.md)
280+
- [Sensors](sensors.md)
281+
- [API Reference](../api/reference.md)

0 commit comments

Comments
 (0)