@@ -616,32 +616,39 @@ orphan files for normal Paimon cleanup.
616616## Load LeRobot Dataset v3
617617
618618` load_from_lerobot ` imports a local directory, FileIO URI, or Hugging Face
619- repository. It derives the schema from ` meta/info.json ` , writes one row per
620- frame, and creates a LeRobot dataset backed by the frame table,
621- ` <table>__versions ` , ` <table>__episodes ` , ` <table>__tasks ` , and an optional
622- ` <table>__subtasks ` . Task text remains in the metadata table; frames retain
623- ` task_index ` . After all components are committed and tagged with the same
624- numeric ` version_id ` , one row is appended to ` <table>__versions ` to publish the
625- version.
619+ repository. It derives the schema from ` meta/info.json ` and writes one row per
620+ frame. The import creates ` <table>__episodes ` , ` <table>__tasks ` , ` <table>__info ` ,
621+ and optional ` <table>__stats ` and ` <table>__subtasks ` companion tables. Task text
622+ remains in the task table; frames retain ` task_index ` .
623+
624+ Info and stats use ` key STRING, value STRING ` : each top-level info property or
625+ stats feature becomes one row. Each value is JSON-encoded, preserving nested
626+ objects, arrays, nulls, and scalar types. Decode it with ` json.loads ` ; for example,
627+ ` fps ` stores ` 30 ` , while ` codebase_version ` stores ` "v3.0" ` (including quotes).
628+ Statistics may contain ` NaN ` and ` Infinity ` , supported by Python's JSON decoder.
629+ Missing or empty stats create no stats table. The frame table's
630+ ` pypaimon.lerobot.<component>-table ` options identify the components created.
626631
627632``` shell
628633pip install ' pypaimon[lerobot]'
629634```
630635
631636``` python
632- version_id = conn.load_from_lerobot(
637+ conn.load_from_lerobot(
633638 " robot_data" ,
634639 " /data/lerobot_dataset" ,
640+ tag_name = " initial-import" , # Optional: tag all imported component snapshots.
635641)
636- print (version_id)
637642```
638643
639- The returned ` version_id ` is the common tag name for all dataset components.
644+ The call returns ` None ` on success. Omitting ` tag_name ` imports the tables
645+ without creating tags. The one-time importer requires a new target table and
646+ a non-empty source dataset; subsequent table edits use the normal Paimon APIs.
640647
641648For FileIO URIs, pass credentials through ` source_options ` :
642649
643650``` python
644- version_id = conn.load_from_lerobot(
651+ conn.load_from_lerobot(
645652 " robot_data" ,
646653 " oss://source-bucket/lerobot_dataset" ,
647654 source_options = {
@@ -652,11 +659,37 @@ version_id = conn.load_from_lerobot(
652659)
653660```
654661
655- A row in ` <table>__versions ` identifies a published release. Readers must first
656- resolve that row, then read every required component through its matching tag;
657- a missing tag is an error and must not fall back to the latest snapshot.
658- The component tags are immutable and must be retained or deleted together.
659- The one-time importer requires a new target table.
662+ Before training, finish any related data/metadata updates and pause writes to
663+ this table group. Create a common named tag over the current component snapshots:
664+
665+ ``` python
666+ tag = " train-2026-09-07"
667+ snapshots = conn.create_lerobot_tag(" robot_data" , tag)
668+ frames = conn.get_table(" robot_data" ).scan(tag_name = tag).to_arrow()
669+
670+ # Companion tables are ordinary Paimon tables, read with the same tag.
671+ info_table = conn.catalog.get_table(" default.robot_data__info" ).copy(
672+ {" scan.tag-name" : tag})
673+ builder = info_table.new_read_builder()
674+ info_rows = builder.new_read().to_arrow(builder.new_scan().plan().splits())
675+
676+ import json
677+ info = {
678+ row[" key" ]: json.loads(row[" value" ])
679+ for row in info_rows.to_pylist()
680+ }
681+ ```
682+
683+ ` create_lerobot_tag ` returns component names mapped to snapshot IDs; these IDs
684+ may differ across tables. Later appends do not change tagged reads. Read every
685+ required component (including training statistics) through the same tag, and
686+ never fall back to latest if a tag is missing.
687+
688+ Cross-table tagging is not atomic. Use a tag only after the creation call
689+ succeeds. A failure may leave partial tags; retry with writes still paused and
690+ unchanged snapshots, or choose a new name after repairing the group. Existing
691+ tags are never moved to different snapshots. Retain or delete component tags
692+ together, and keep writers paused until the call returns.
660693
661694Scalars map to scalar types, vectors to ` VECTOR ` , higher-rank tensors to nested
662695` ARRAY ` , and images to ` BLOB ` . Images keep their compressed bytes.
0 commit comments