Repository navigation
[59_maintenance] Write empty validity buffers for arrays without nulls in IPC writer - #11820
Merged
Merged
Conversation
4 of 5 tasks
github-actions
Bot
added
arrow-flight
…pache#11694) # Which issue does this PR close? N/A # Rationale for this change The IPC writer writes a validity bitmap for every array that has a validity buffer, also when the array has no nulls. If the array has no null buffer, `write_array_data` creates an all-ones bitmap of `ceil(len / 8)` bytes for it. Readers do not use it: the arrow-rs reader, like the C++ reader, ignores the validity buffer when `null_count` is 0. The format allows a 0-length buffer instead. [`FieldNode.null_count` in `Message.fbs`](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/format/Message.fbs#L39-L41): > Fields with null_count == 0 may choose not to write their physical validity bitmap out as a materialized buffer, instead setting the length of the bitmap buffer to 0. The [C++](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/arrow/ipc/writer.cc#L178-L187), [Go](https://github.com/apache/arrow-go/blob/a6e44ad7a6aaa4c02a7d277032af8eebd5cc2508/arrow/ipc/writer.go#L634-L638) and [JS](https://github.com/apache/arrow-js/blob/43c9547baf3766e79eb44d3fedcd0787f899d956/src/visitor/vectorassembler.ts#L95-L98) writers already do this. So arrow-rs already reads such buffers, in the integration tests and in the `arrow-testing` gold files, which C++ wrote. # What changes are included in this PR? - `write_array_data` writes a 0-length validity buffer when `null_count` is 0. This also covers sliced arrays whose null buffer has no nulls in the slice. Arrays with nulls are written as before. - `test_decimal128_alignment8_is_unaligned` relied on the all-ones bitmap to move the decimal values to an 8-byte offset. It now uses a nullable column with a null, so it gets the expected misalignment error with both the old and the new writer. - The exact overage limits in the arrow-flight `verify_encoded_split` tests go down, for example from 120 to 88. # Are these changes tested? Yes. `test_write_empty_validity_buffer_without_nulls` encodes a batch with three columns: an array without a null buffer, a slice without nulls that keeps its null buffer, and an array with a null. It checks the validity buffer lengths in the encoded message (`[0, 0, 1]`) and the round trip. Without the change, the first two columns get 1-byte bitmaps and the test fails. # Are there any significant user-facing changes? The IPC output for arrays without nulls is smaller. The spec allows this, and the C++, Java, Go, JS, C# and nanoarrow readers accept it. Code that compares IPC output byte for byte, or expects exact sizes, will see a difference. There are no API changes. (cherry picked from commit 49219f8)
alamb
force-pushed
the
backport_11694_59
branch
from
October 22, 2026 15:33
0e1da6e to
1016566
Compare
4 of 5 tasks
alamb
added a commit
to alamb/arrow-rs
that referenced
this pull request
Oct 22, 2026
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
59.3.2(October 2026) #1166458.4.1(October 2026) #11665 (comment))Cherry-pick of 49219f8
The cherry-pick applied cleanly but did not compile: on main
write_array_datahas anull_countlocal introduced by #10128, which is not on59_maintenance. This PR adds that one-line local (let null_count = array_data.null_count();) so the backported code matches main.