Repository navigation
[59_maintenance] Parquet: list the dictionary page first in page_encoding_stats - #11699
Merged
Merged
Conversation
…e#11682) - Closes apache#11659 Since 59.0.0, `ArrowWriter` writes the dictionary page's `encoding_stats` entry after the data page entries. The file layout did not change: the dictionary page is still written first. Arrow C++ ([`IsColumnChunkFullyDictionaryEncoded`](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/parquet/file_reader.cc#L63-L88)) requires `encoding_stats[0]` to be the dictionary page. With the 59.x order, Arrow C++ does not see any parquet-rs column chunk as fully dictionary encoded, so it cannot use its dictionary fast path. | | `encoding_stats` (file order) | |---|---| | before this PR | `DATA_PAGE RLE_DICTIONARY`, `DICTIONARY_PAGE PLAIN` | | after this PR (and 58.x) | `DICTIONARY_PAGE PLAIN`, `DATA_PAGE RLE_DICTIONARY` | The new order was not a design goal of apache#10020. It was a side effect: the deferred-ordering page writer gets the dictionary page last, and the stats were recorded in the order the pages were produced. Review accepted this because the spec defines the list as a set (apache#10051 was closed for that reason). But a known reader depends on the order, and restoring it changes only metadata, so the change has no memory or throughput cost. `GenericColumnWriter::write_dictionary_page` now inserts the dictionary entry at the front of `encoding_stats` instead of appending it. The dictionary page is always laid out first, on both the deferred (Arrow) path and the `SerializedFileWriter` path, so the stats now follow file order. On the non-deferred path the list is still empty when the dictionary is written, so its output does not change. When the writer falls back from dictionary to plain encoding, the result is `DICTIONARY, RLE_DICTIONARY data, PLAIN data`, which also matches the file layout. Yes. The new `dictionary_page_encoding_stats_lists_dictionary_first` test (adapted from apache#10051) checks that `stats[0]` is the dictionary page. It fails on `main` and passes with this change. The existing column writer and arrow writer tests pass. Only the order of `page_encoding_stats` in files written by `ArrowWriter`. It goes back to the order used before 59.0.0. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
59.3.2(October 2026) #11664Cherry-pick of 33d338d