Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions cookbooks/cosmos3/generator/action/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,33 @@ fingers.

Action data samples across different embodiments can be inspected interactively in the [Cosmos3 Action Viewer](https://huggingface.co/spaces/nvidia/Cosmos3-Action-Viewer) Hugging Face Space.

### Dataset state, action layout, and normalization

The inverse-dynamics examples predict action trajectories from video inputs.
They do not infer a full LeRobot episode by themselves: fields such as
`observation.state`, timestamps, frame indexes, and camera streams must come
from the source dataset or the robot logging pipeline used to create the
episode. When building a LeRobot-format dataset, align the predicted `action`
rows with the original per-timestep observations instead of treating inverse
dynamics as a replacement for state estimation.

For a concrete LeRobot-style robotics sample, inspect
[`assets/droid_lerobot_example/`](./assets/droid_lerobot_example/). Its
metadata declares the DROID state streams
`observation.state.cartesian_position`,
`observation.state.joint_positions`, and
`observation.state.gripper_position`, while the parquet/video assets provide the
corresponding timestep-aligned records. This is the current checked-in example
for understanding how action-conditioned robotics inputs relate to dataset
state fields.

Use the action-definition table above as the current layout summary for the
checked-in AV, DROID, UMI, and human hand-pose examples. The corresponding
action JSON files and notebooks show how those layouts are normalized and passed
to each backend. Other embodiment layouts and normalization statistics should be
treated as model/data-release specific until their example assets or framework
configuration are published.

## Run with Cosmos Framework

### Quickstart
Expand Down