diff --git a/cookbooks/cosmos3/generator/action/README.md b/cookbooks/cosmos3/generator/action/README.md index 51ea8268..04d8dab1 100644 --- a/cookbooks/cosmos3/generator/action/README.md +++ b/cookbooks/cosmos3/generator/action/README.md @@ -58,6 +58,33 @@ fingers. Action data samples across different embodiments can be inspected interactively in the [Cosmos3 Action Viewer](https://huggingface.co/spaces/nvidia/Cosmos3-Action-Viewer) Hugging Face Space. +### Dataset state, action layout, and normalization + +The inverse-dynamics examples predict action trajectories from video inputs. +They do not infer a full LeRobot episode by themselves: fields such as +`observation.state`, timestamps, frame indexes, and camera streams must come +from the source dataset or the robot logging pipeline used to create the +episode. When building a LeRobot-format dataset, align the predicted `action` +rows with the original per-timestep observations instead of treating inverse +dynamics as a replacement for state estimation. + +For a concrete LeRobot-style robotics sample, inspect +[`assets/droid_lerobot_example/`](./assets/droid_lerobot_example/). Its +metadata declares the DROID state streams +`observation.state.cartesian_position`, +`observation.state.joint_positions`, and +`observation.state.gripper_position`, while the parquet/video assets provide the +corresponding timestep-aligned records. This is the current checked-in example +for understanding how action-conditioned robotics inputs relate to dataset +state fields. + +Use the action-definition table above as the current layout summary for the +checked-in AV, DROID, UMI, and human hand-pose examples. The corresponding +action JSON files and notebooks show how those layouts are normalized and passed +to each backend. Other embodiment layouts and normalization statistics should be +treated as model/data-release specific until their example assets or framework +configuration are published. + ## Run with Cosmos Framework ### Quickstart