Skip to content

docs: clarify action state and normalizer expectations - #212

Open
Rohithmatham12 wants to merge 1 commit into
NVIDIA:mainfrom
Rohithmatham12:clarify-action-state-docs
Open

docs: clarify action state and normalizer expectations#212
Rohithmatham12 wants to merge 1 commit into
NVIDIA:mainfrom
Rohithmatham12:clarify-action-state-docs

Conversation

@Rohithmatham12

Copy link
Copy Markdown

Summary

  • clarify that inverse dynamics predicts action trajectories but does not synthesize full LeRobot observation.state records
  • point users to the checked-in DROID LeRobot sample metadata for timestep-aligned state streams
  • document that current action-layout and normalization references are the checked-in AV, DROID, and UMI assets/notebooks until additional embodiment assets are released

Why
Issue #192 asks how to obtain observation.state alongside predicted actions for LeRobot-format episodes. Issue #184 asks for action channel layouts and normalizers across embodiments. This PR documents the current supported boundary and points to the concrete checked-in sample without claiming unreleased embodiment-specific layouts.

Testing

  • git diff --check

Addresses #192
Related to #184

@mli0603

mli0603 commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

Thanks @Rohithmatham12. Will consolidate the your PR and the docs! We have a PR landing soon and we will take a look.

@MaciejBalaNV

Copy link
Copy Markdown
Collaborator

@mli0603 @Rohithmatham12
Is this PR still relevant? There were a few changes recently in the action normalization on cosmos_framework side.

@Rohithmatham12
Rohithmatham12 force-pushed the clarify-action-state-docs branch from 4870139 to fda1c0b Compare August 23, 2026 02:29
@Rohithmatham12

Rohithmatham12 commented Aug 23, 2026

Copy link
Copy Markdown
Author

Thanks for checking. I rebased this on current main and refreshed the normalization wording.

I think the PR is still relevant for the observation.state / LeRobot part of #192: inverse dynamics returns predicted action trajectories, but it still does not synthesize a full LeRobot episode record, so users need to align predicted action rows with the source dataset's timestep-level observation.state, frame, timestamp, and camera streams.

For the normalization side, I updated the new section so it points at the current action-definition table on this README as the layout summary for AV, DROID, UMI, and human hand-pose examples, with the JSON assets/notebooks showing how those layouts are normalized and passed to each backend. It no longer implies the older checked-in assets are the only reference after the cosmos_framework changes.

If you prefer to fold this into a broader docs update, that is fine too, but I think the state-vs-action clarification is still a useful boundary to keep documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants