[Bug] parquet files with columns containing large list of byte arrays can not be read by pyarrow.

[daw3rd]

Search before asking

• I searched the issues and found no similar issues.

Component

Library/core

What happened + What you expected to happen

I have a parquet file that has a column containing a list of images, as byte arrays. Under some circumstances, such files are not readable by pyarrow.

Reproduction script

grab https://ibm.ent.box.com/file/1684883605503?s=9qcne0iubeji6t6a77gh2sxgi29k1tjp as test.parquet

cd transforms/universal/noop/python
mkdir input
cp .../test.parquet input
make venv
source venv/bin/activate
python src/noop_transform_python.py --data_local_config  "{'input_folder': 'input', 'output_folder':'output'}"
09:37:35 INFO - Launching noop transform
09:37:35 INFO - noop parameters are : {'sleep_sec': 1, 'pwd': 'nothing'}
09:37:35 INFO - pipeline id pipeline_id
09:37:35 INFO - code location None
09:37:35 INFO - data factory data_ is using local data access: input_folder - input output_folder - output
09:37:35 INFO - data factory data_ max_files -1, n_sample -1
09:37:35 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']
09:37:35 INFO - orchestrator noop started at 2024-11-20 09:37:35
09:37:35 INFO - Number of files is 1, source profile {'max_file_size': 341.400185585022, 'min_file_size': 341.400185585022, 'total_file_size': 341.400185585022}
09:37:39 ERROR - Failed to convert byte array to arrow table, exception Nested data conversions not implemented for chunked array outputs. Skipping it
09:37:39 WARNING - Transformation of file to table failed
09:37:39 INFO - Completed 1 files (100.0%) in 0.07 min
09:37:39 INFO - Done processing 1 files, waiting for flush() completion.
09:37:39 INFO - done flushing in 0.0 sec
09:37:39 INFO - Completed execution in 0.07 min, execution result 0
Segmentation fault: 11

Anything else

(venv) dawood@MacBookPro:~/git/data-prep-kit/transforms/universal/noop/python$ parquet-tools inspect input/test.parquet 

############ file meta data ############
created_by: parquet-cpp-arrow version 16.1.0
num_columns: 16
num_rows: 1000
num_row_groups: 1
format_version: 2.6
serialized_size: 4643


############ Columns ############
id
contents
source
element
from
value
orig_json_cos_path
element
index
element
element
nfaces
nsfw
nsfw_score
prompted_child_score
prompted_nsfw_score

############ Column(id) ############
name: id
path: id
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 82%)

############ Column(contents) ############
name: contents
path: contents
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 51%)

############ Column(source) ############
name: source
path: source
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: -7%)

############ Column(element) ############
name: element
path: orig_image_fpaths.list.element
max_definition_level: 3
max_repetition_level: 1
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 86%)

############ Column(from) ############
name: from
path: conversations.list.element.from
max_definition_level: 4
max_repetition_level: 1
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 81%)

############ Column(value) ############
name: value
path: conversations.list.element.value
max_definition_level: 4
max_repetition_level: 1
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 50%)

############ Column(orig_json_cos_path) ############
name: orig_json_cos_path
path: orig_json_cos_path
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 1%)

############ Column(element) ############
name: element
path: image_bins.list.element
max_definition_level: 3
max_repetition_level: 1
physical_type: BYTE_ARRAY
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 0%)

############ Column(index) ############
name: index
path: index
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 43%)

############ Column(element) ############
name: element
path: fixed_image_fpaths.list.element
max_definition_level: 3
max_repetition_level: 1
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 81%)

############ Column(element) ############
name: element
path: blurred_images.list.element
max_definition_level: 3
max_repetition_level: 1
physical_type: BYTE_ARRAY
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 0%)

############ Column(nfaces) ############
name: nfaces
path: nfaces
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: -4%)

############ Column(nsfw) ############
name: nsfw
path: nsfw
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: -4%)

############ Column(nsfw_score) ############
name: nsfw_score
path: nsfw_score
max_definition_level: 1
max_repetition_level: 0
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 14%)

############ Column(prompted_child_score) ############
name: prompted_child_score
path: prompted_child_score
max_definition_level: 1
max_repetition_level: 0
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 11%)

############ Column(prompted_nsfw_score) ############
name: prompted_nsfw_score
path: prompted_nsfw_score
max_definition_level: 1
max_repetition_level: 0
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 12%)

OS

Ubuntu, MacOS (limited support)

Python

3.11.x

Are you willing to submit a PR?

• Yes I am willing to submit a PR!