feat: Add read support for Parquet bloom filters #2653
Conversation
…et bloom filters. Closes apache#2649
There was a problem hiding this comment.
Thanks @ForeverAngry for working on this. Since this changes the specification, we have to go through an Iceberg improvement proposal to ensure that there is concensus across different implementations.
As part of the change process, my main question would be; what's the added value on top the bloom filters that are embedded in the Parquet files.
| NestedField( | ||
| field_id=146, | ||
| name="bloom_filter_bytes", | ||
| field_type=MapType(key_id=147, key_type=IntegerType(), value_id=148, value_type=BinaryType()), | ||
| required=False, | ||
| doc="Map of column id to bloom filter", | ||
| ), |
There was a problem hiding this comment.
We cannot just add a field; this requires a spec change: https://iceberg.apache.org/contribute/#apache-iceberg-improvement-proposals
|
@Fokko I was kinda thinking that when I submitted it. But, it was done, and I figured id just send it to see if it sparked any interest. That's good information though, sometimes I forget about the governance structures that exist for these projects. |
I guess, to me, the main benefit would be the ability to do file-level pruning before opening any files. As a result, this would also come with some secondary benefits like being able to do row group-level pruning within a Parquet file after opening it. |
2 participants
Closes #2649
Rationale for this change
Add support for using bloom filters in the read path of pyiceberg.
Are these changes tested?
Yes.
Are there any user-facing changes?
I dont think so.