Repository navigation
Return extension values from ExtensionStoragePushdown - #11188
Closed
connortsui20 wants to merge 2 commits into
Closed
connortsui20 wants to merge 2 commits into
connortsui20 wants to merge 2 commits into
Conversation
`ExtensionStoragePushdown` lent its own return and partial dtypes to the aggregate that it runs on storage. The next commit gives the fields of those dtypes that hold input values the extension dtype again, so the storage aggregate needs dtypes of its own, resolved over the storage dtype. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
`ExtensionStoragePushdown` returned the results of its aggregate over storage values unchanged. So `vortex.min_max[vortex.timestamp]` gave raw `i64` values without the time unit or zone, while plain `min_max` gives timestamps. The new `StorageResultMapping` trait names the value fields of a partial and a result, which hold input values, and `MinMax` and `IsSorted` implement it. The pushdown requires the trait, so an aggregate without a mapping does not compile. Partial states stay in storage form in memory. `to_scalar` and `finalize_scalar` give their output the extension dtype, `partial_from_scalar` removes it, and `finalize` receives states that already have it. The tests now compare the pushdown with plain `min_max` over flat, chunked, constant, dictionary-encoded, and grouped input, and check that an `is_sorted` partial with a null first value survives its round trip through a scalar. The test that registered the pushdown by hand is gone, because the default session test already checks the registration. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
marked this pull request as ready for review
November 3, 2026 22:55
connortsui20
marked this pull request as draft
November 3, 2026 22:56
Merging this PR will degrade performance by 0.81%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
10 of 47 tasks
connortsui20
added a commit
that referenced
this pull request
Nov 4, 2026
TS (#10953) merged on 2026-11-03, and SN1 (#11183) and V10 (#11187) merged on 2026-11-04. The dependency graph marks the three merged and moves its state date to 2026-11-04. It gives SD, DP, FD and U9 their open PRs (#11184, #11185, #11186 and #11188), and it drops #10408, which closed without a merge. The text of parts 4 and 5 and the Fix lines of EXT-V9, EXT-V10, EXT-N2 and EXT-S3 now record the same state. Also record that AX, DS and SN2 are implemented on branches without a PR, and add the approved one-pass design for the masked nulls of SN2. Part 2 and decision X31 also record the fixes: #10953 made every i64 legal, #11183 made the spatial validators reject nullable nested levels, #11187 gave the Date display its fallback, and #10408 closed. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
added a commit
that referenced
this pull request
Nov 4, 2026
## Summary Every grouped `is_sorted` and `is_constant` call fails with "Return DType mismatch". Both `finalize` methods read the result field out of the nullable partial struct, which gives `Bool?` while the return dtype is a non-nullable `Bool`. #11188 found the `is_sorted` case, and its grouped pushdown test covers only `min_max` because of it. ## Changes Both `finalize` methods now fill null partials. A null partial comes from an empty or null group and is the empty state, so each fill uses the verdict that `finalize_scalar` already gives that state: `true` for `is_sorted` (vacuously sorted) and `false` for `is_constant`. Grouped and scalar results now agree. Note that a null group gets the same verdict as an empty group, since the partial can't tell them apart and the return dtype is non-nullable. Each aggregate gets a grouped test that puts empty, all-null, and null groups next to ordinary ones. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Member
Author
connortsui20
added a commit
that referenced
this pull request
Nov 4, 2026
The maintainer closed #11188 on 2026-11-04, because naming the value fields by index is a half-measure and SP4 removes the adapter anyway. Issue #11243 tracks the bug until SP4. X12, part 4, the plan, and the findings now say so, and part 3 asks whether SP1 needs a positional `Fields` mapping at all. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ExtensionStoragePushdownreturned the results of its aggregate over storage as they were, sovortex.min_max[vortex.timestamp]gave rawi64values without the time unit or time zone. Plainmin_maxreturns timestamps for the same column, and nothing restores the type later, so the pushdown now returns extension values too.Changes
Through a new
StorageResultMappingtrait,MinMaxandIsSortedname their value fields, the fields of their partials and results that hold input values. They name them by index, since a dtype can't tell them apart: overBoolstorage, theis_sortedfield also has the storage dtype. Groupedfinalizereceives partials whose value fields already hold extension values, so the trait also requires thatfinalizeonly moves or drops those fields and ignoresargs.dtype, as both aggregates already do.The fixed mapping is a stopgap until #11115 gives every aggregate that runs on storage a result mapping through
ExtensionUse::Storage. The impls sit next to the aggregates, so #10769 can move them along.Tests compare the pushdown with plain
min_maxover flat, chunked, constant, dictionary-encoded, and grouped input, and check that anis_sortedpartial with a null value survives its round trip through a scalar.API Changes
ExtensionStoragePushdown<A, E>now requiresA: StorageResultMapping, which onlyMinMaxandIsSortedimplement. The value fields of the results and partials, such asminandmaxofvortex.min_max[vortex.timestamp], now have the extension dtype, and a storage value that the extension type rejects is an error, as with plainmin_max.