Skip to content

Commit 12e030f

Browse files
YannByronclaudeCodex
committed
feat(python): simplify LeRobot metadata tables and training tags
Store info and stats in companion tables with JSON string values, and support optional named tags for imported or current component snapshots. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-6 Co-Authored-By: Codex <noreply@ai-tool.com> AI-Contributed/Feature: 22/22 AI-Contributed/UT: 38/38
1 parent e7db7cf commit 12e030f

6 files changed

Lines changed: 392 additions & 180 deletions

File tree

‎docs/docs/pypaimon/multimodal-api.mdx‎

Lines changed: 49 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -616,32 +616,39 @@ orphan files for normal Paimon cleanup.
616616
## Load LeRobot Dataset v3
617617

618618
`load_from_lerobot` imports a local directory, FileIO URI, or Hugging Face
619-
repository. It derives the schema from `meta/info.json`, writes one row per
620-
frame, and creates a LeRobot dataset backed by the frame table,
621-
`<table>__versions`, `<table>__episodes`, `<table>__tasks`, and an optional
622-
`<table>__subtasks`. Task text remains in the metadata table; frames retain
623-
`task_index`. After all components are committed and tagged with the same
624-
numeric `version_id`, one row is appended to `<table>__versions` to publish the
625-
version.
619+
repository. It derives the schema from `meta/info.json` and writes one row per
620+
frame. The import creates `<table>__episodes`, `<table>__tasks`, `<table>__info`,
621+
and optional `<table>__stats` and `<table>__subtasks` companion tables. Task text
622+
remains in the task table; frames retain `task_index`.
623+
624+
Info and stats use `key STRING, value STRING`: each top-level info property or
625+
stats feature becomes one row. Each value is JSON-encoded, preserving nested
626+
objects, arrays, nulls, and scalar types. Decode it with `json.loads`; for example,
627+
`fps` stores `30`, while `codebase_version` stores `"v3.0"` (including quotes).
628+
Statistics may contain `NaN` and `Infinity`, supported by Python's JSON decoder.
629+
Missing or empty stats create no stats table. The frame table's
630+
`pypaimon.lerobot.<component>-table` options identify the components created.
626631

627632
```shell
628633
pip install 'pypaimon[lerobot]'
629634
```
630635

631636
```python
632-
version_id = conn.load_from_lerobot(
637+
conn.load_from_lerobot(
633638
"robot_data",
634639
"/data/lerobot_dataset",
640+
tag_name="initial-import", # Optional: tag all imported component snapshots.
635641
)
636-
print(version_id)
637642
```
638643

639-
The returned `version_id` is the common tag name for all dataset components.
644+
The call returns `None` on success. Omitting `tag_name` imports the tables
645+
without creating tags. The one-time importer requires a new target table and
646+
a non-empty source dataset; subsequent table edits use the normal Paimon APIs.
640647

641648
For FileIO URIs, pass credentials through `source_options`:
642649

643650
```python
644-
version_id = conn.load_from_lerobot(
651+
conn.load_from_lerobot(
645652
"robot_data",
646653
"oss://source-bucket/lerobot_dataset",
647654
source_options={
@@ -652,11 +659,37 @@ version_id = conn.load_from_lerobot(
652659
)
653660
```
654661

655-
A row in `<table>__versions` identifies a published release. Readers must first
656-
resolve that row, then read every required component through its matching tag;
657-
a missing tag is an error and must not fall back to the latest snapshot.
658-
The component tags are immutable and must be retained or deleted together.
659-
The one-time importer requires a new target table.
662+
Before training, finish any related data/metadata updates and pause writes to
663+
this table group. Create a common named tag over the current component snapshots:
664+
665+
```python
666+
tag = "train-2026-09-07"
667+
snapshots = conn.create_lerobot_tag("robot_data", tag)
668+
frames = conn.get_table("robot_data").scan(tag_name=tag).to_arrow()
669+
670+
# Companion tables are ordinary Paimon tables, read with the same tag.
671+
info_table = conn.catalog.get_table("default.robot_data__info").copy(
672+
{"scan.tag-name": tag})
673+
builder = info_table.new_read_builder()
674+
info_rows = builder.new_read().to_arrow(builder.new_scan().plan().splits())
675+
676+
import json
677+
info = {
678+
row["key"]: json.loads(row["value"])
679+
for row in info_rows.to_pylist()
680+
}
681+
```
682+
683+
`create_lerobot_tag` returns component names mapped to snapshot IDs; these IDs
684+
may differ across tables. Later appends do not change tagged reads. Read every
685+
required component (including training statistics) through the same tag, and
686+
never fall back to latest if a tag is missing.
687+
688+
Cross-table tagging is not atomic. Use a tag only after the creation call
689+
succeeds. A failure may leave partial tags; retry with writes still paused and
690+
unchanged snapshots, or choose a new name after repairing the group. Existing
691+
tags are never moved to different snapshots. Retain or delete component tags
692+
together, and keep writers paused until the call returns.
660693

661694
Scalars map to scalar types, vectors to `VECTOR`, higher-rank tensors to nested
662695
`ARRAY`, and images to `BLOB`. Images keep their compressed bytes.

‎paimon-python/README.md‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,17 +44,29 @@ pip install 'pypaimon[lerobot]'
4444
import pypaimon.multimodal as pmm
4545

4646
connection = pmm.connect(options={"warehouse": "/tmp/warehouse"})
47-
version_id = connection.load_from_lerobot(
47+
connection.load_from_lerobot(
4848
"robot_data",
4949
"/data/lerobot_dataset",
5050
)
51-
print(version_id)
5251
```
5352

5453
The source dataset must be non-empty. Its schema comes from `meta/info.json`.
5554
Each frame becomes one row; media uses BLOB columns. The import creates frame,
56-
Episode, task, and version tables and tags the three component tables with the
57-
returned `version_id`.
55+
Episode, task, info, and optional stats/subtask tables. Info and stats use
56+
`key STRING, value STRING` rows, with each value JSON-encoded to preserve
57+
nested metadata. Decode values with `json.loads`.
58+
59+
Before training, pause writes and create a shared tag:
60+
61+
```python
62+
connection.create_lerobot_tag("robot_data", "train-2026-09-07")
63+
frames = connection.get_table("robot_data").scan(
64+
tag_name="train-2026-09-07").to_arrow()
65+
```
66+
67+
Read every metadata component with the same tag. Use the tag only after creation
68+
succeeds; cross-table tagging is not atomic. Alternatively, pass `tag_name` to
69+
`load_from_lerobot` to tag the imported snapshots immediately.
5870

5971
# HDF5 to multimodal tables
6072

‎paimon-python/pypaimon/multimodal/connection.py‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -141,18 +141,29 @@ def load_from_lerobot(
141141
*,
142142
batch_size: int = 1024,
143143
options=None,
144-
source_options=None):
144+
source_options=None,
145+
tag_name=None) -> None:
145146
"""Import LeRobot Dataset v3 into a new Paimon table group."""
146147
from pypaimon.multimodal.lerobot import load_from_lerobot
147-
return load_from_lerobot(
148+
load_from_lerobot(
148149
self,
149150
table_name,
150151
source,
151152
batch_size=batch_size,
152153
options=options,
153154
source_options=source_options,
155+
tag_name=tag_name,
154156
)
155157

158+
def create_lerobot_tag(self, table_name: str, tag_name: str):
159+
"""Pin all LeRobot components; pause group writes until this returns.
160+
161+
Returns component snapshot IDs. Use the tag only after success and
162+
retain it on every component for the lifetime of a training run.
163+
"""
164+
from pypaimon.multimodal.lerobot.metadata import create_lerobot_tag
165+
return create_lerobot_tag(self, table_name, tag_name)
166+
156167
def load_from_rosbag(
157168
self,
158169
table_name: str,

‎paimon-python/pypaimon/multimodal/lerobot/api.py‎

Lines changed: 22 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,14 @@
2222

2323
from pypaimon.catalog.catalog_exception import TableAlreadyExistException
2424
from pypaimon.multimodal.lerobot.metadata import (
25+
_COMPANION_OPTION_KEYS,
2526
_append_arrow_tables,
2627
_load_dataset_metadata,
2728
_managed_table_options,
2829
_prepare_metadata_tables,
2930
_positive_integer,
30-
_publish_dataset,
31+
_commit_metadata,
32+
_validate_tag_name,
3133
_validated_episode_tables,
3234
)
3335
from pypaimon.multimodal.lerobot.loader import _write_dataset
@@ -58,11 +60,13 @@ def load_from_lerobot(
5860
batch_size: int = 1024,
5961
options: Optional[Mapping[str, object]] = None,
6062
source_options: Optional[Mapping[str, object]] = None,
61-
) -> int:
62-
"""Import LeRobot Dataset v3 and return its version ID.
63+
tag_name: Optional[str] = None,
64+
) -> None:
65+
"""Import LeRobot Dataset v3 into a new Paimon table group.
6366
6467
A new target table is created from LeRobot metadata. Episode, task, and
65-
version metadata are stored in companion Paimon tables.
68+
info/stats metadata are stored in companion Paimon tables. If provided,
69+
``tag_name`` pins all components to their imported snapshots.
6670
FileIO URI credentials come only from ``source_options`` and are not
6771
inherited from the target Catalog.
6872
"""
@@ -74,6 +78,9 @@ def load_from_lerobot(
7478
or batch_size <= 0:
7579
raise ValueError("batch_size must be a positive integer.")
7680

81+
if tag_name is not None:
82+
_validate_tag_name(tag_name)
83+
7784
validated_source_options = _validated_source_options(source_options)
7885
_validate_source_kerberos(
7986
[source], validated_source_options, "LeRobot")
@@ -99,7 +106,7 @@ def load_from_lerobot(
99106
lerobot_schema = _schema_from_info(info)
100107
metadata = _load_dataset_metadata(
101108
dataset, info, resolved_source)
102-
return _import_dataset(
109+
_import_dataset(
103110
connection,
104111
table_name,
105112
dataset,
@@ -109,6 +116,7 @@ def load_from_lerobot(
109116
batch_size,
110117
options,
111118
metadata,
119+
tag_name,
112120
)
113121
finally:
114122
close = getattr(dataset, "close", None)
@@ -125,12 +133,12 @@ def _import_dataset(
125133
source_schema,
126134
batch_size,
127135
options,
128-
metadata):
136+
metadata,
137+
tag_name):
129138
table = _create_target_table(
130-
connection, table_name, source_schema, options)
139+
connection, table_name, source_schema, options, metadata)
131140
tables = _prepare_metadata_tables(
132141
connection, table.raw_table, metadata)
133-
version_id = 1
134142
episodes_snapshot_id = _append_arrow_tables(
135143
tables["episodes"],
136144
_validated_episode_tables(metadata),
@@ -146,16 +154,15 @@ def _import_dataset(
146154
batch_size,
147155
metadata,
148156
)
149-
_publish_dataset(
157+
_commit_metadata(
150158
connection,
151159
tables,
152-
version_id,
160+
tag_name,
153161
metadata,
154162
table.identifier,
155163
frames_snapshot_id,
156164
episodes_snapshot_id,
157165
)
158-
return version_id
159166

160167

161168
def _validated_counts(info, source):
@@ -188,11 +195,12 @@ def _required_count(info, name, source):
188195

189196

190197
def _create_target_table(
191-
connection, table_name, source_schema, options):
198+
connection, table_name, source_schema, options, metadata):
192199
create_options = dict(options or {})
193200
managed_options = _managed_table_options(
194-
connection._identifier(table_name))
195-
reserved_options = set(managed_options).intersection(create_options)
201+
connection._identifier(table_name), metadata)
202+
reserved_options = set(_COMPANION_OPTION_KEYS.values()).intersection(
203+
create_options)
196204
if reserved_options:
197205
raise ValueError(
198206
"%s are managed by load_from_lerobot."

0 commit comments

Comments
 (0)