@@ -905,6 +905,68 @@ Notes:
905905 few large reads); scattered point reads coalesce less.
906906- Blob reads are available only on ` scan() ` , not on the ` search() ` queries.
907907
908+ ### Contiguous windows for PyTorch
909+
910+ Install the ` torch ` extra, then use ` to_contiguous_window_dataset ` to expose
911+ map-style windows without loading the selected rows or BLOB payloads into Python
912+ memory up front. The Dataset builds a compact index from the group column, order
913+ column, and Paimon row IDs. Each ` __getitem__ ` call fetches only that window from
914+ the snapshot recorded in ` dataset.snapshot_id ` .
915+
916+ ``` shell
917+ pip install ' pypaimon[torch]'
918+ ```
919+
920+ ``` python
921+ import torch
922+
923+
924+ def float32_window (values ):
925+ return torch.tensor(values, dtype = torch.float32)
926+
927+
928+ windows = (
929+ frames.scan()
930+ .where(" split = 'train'" )
931+ .to_contiguous_window_dataset(
932+ window_size = 16 ,
933+ columns = [" state" , " action" ],
934+ group_key = " episode_index" ,
935+ order_key = " frame_index" ,
936+ tail = " pad" ,
937+ column_transforms = {
938+ " state" : float32_window,
939+ " action" : float32_window,
940+ },
941+ )
942+ )
943+
944+ sample = windows[0 ]
945+ assert sample[" action" ].shape == (16 , action_size)
946+ assert sample[" is_pad" ].shape == (16 ,)
947+ ```
948+
949+ The group and order keys in a sample identify the window anchor. Every projected
950+ column contains the whole window. With ` tail="drop" ` , only full windows are
951+ exposed. With ` tail="pad" ` , every real row is an anchor; missing suffix values
952+ repeat the last real value by default and ` is_pad ` is ` True ` exactly at those
953+ positions. With ` tail="error" ` , construction fails if any scheduled anchor is
954+ incomplete. Use ` pad_values ` to override the repeated value for individual
955+ columns. Anchors advance by ` stride ` , which defaults to one row.
956+
957+ ` column_transforms ` receive one padded Python list per projected column. This is
958+ where applications define tensor dtype and shape or decode BLOB bytes. The
959+ optional ` adapter ` receives the resulting sample mapping and can rename or
960+ combine fields for a model-specific batch contract. The core Dataset does not
961+ know model field names, image formats, or normalization rules. Top-level
962+ functions and callable classes are recommended for transforms and adapters so
963+ the Dataset remains picklable by multi-worker ` torch.utils.data.DataLoader `
964+ instances.
965+
966+ Columns configured by ` video-frame-field ` are rejected: a window read would drop
967+ the ` frame_index ` and other metadata carried by their ` VideoFrameDescriptor `
968+ values. Read those columns with ` to_torch() ` instead.
969+
908970### Distributed BLOB processing with Ray
909971
910972For larger jobs, read descriptors with ` to_ray() ` , then fetch and process BLOB
0 commit comments