[jiengup]

Summary

• Tracking Issue: Migrate every encoding from OperationsVTable::scalar_at to probe_scalar #9971

OperationsVTable::scalar_at cannot retain preparation between row reads. The probe API introduced in #9843 retains encoding state, validity, and child probes across repeated reads.

This PR migrates 30 implementations to explicit probe_scalar methods: eight simple implementations, ten structural implementations, Map, Patched, PiecewiseSequence, ScalarFn, ALPRD, BitPacked, Delta, OnPair, ParquetVariant, Sparse, Zstd, and ZstdBuffers. The simple, structural, Map, ScalarFn, ParquetVariant, and Sparse implementations use ProbeState = () and reuse generic validity and child probes where needed. Patched retains its most recently accessed lane; PiecewiseSequence retains a bounded lookup index; ALPRD retains one patch query; BitPacked retains one validated block and its base boundary; Delta retains one decoded chunk; OnPair retains one row's code boundaries; Zstd retains one bounded decoded window; ZstdBuffers retains one decoded inner probe under a byte cap. Each migrated scalar_at delegates to a one-off probe during the transition.

The current scope is the sixteen commits in origin/develop..HEAD, from 18995b8d91 through cf5a9da2aa, based on origin/develop at d270200d65.

Changes

Milestones

The branch contains separate commits for the simple, structural, Map, Patched, PiecewiseSequence, ScalarFn, ALPRD, BitPacked, Delta, OnPair, ParquetVariant, Sparse, Zstd, and ZstdBuffers milestones, plus a lint and formatting fix and a benchmark coverage commit. The array migrations are complete; the later API removals remain pending below.

• Migration completed
• Make probe_scalar required; deprecate and remove OperationsVTable::scalar_at
• Address the public ArrayRef scalar and validity accessors on the later-release schedule in Migrate every encoding from OperationsVTable::scalar_at to probe_scalar #9971

ScalarFnValidity is an additional internal target. TransposedBool is listed in the issue but has no implementation in the current tree.

Preparation and state

() means no encoding-owned cache; a repeated probe can still retain validity and child probes. Pending designs are marked TBD. Candidate reuse paths are listed to guide each milestone, not to commit to a cache before its benchmark and correctness review.

Encoding	Preparation to reuse	ProbeState	Status
Bool	Generic retained validity	()	Done
Constant	None; the stored scalar carries nullness	()	Done
Decimal	Generic retained validity	()	Done
Null	None; every value is null	()	Done
VarBinView	Generic retained validity	()	Done
ByteBool	Generic retained validity	()	Done
Sequence	None; values are computed directly	()	Done
PythonVTable	None; scalar access remains unsupported	()	Done
Filter	Source child probe at the mask-ranked row	()	Done
Interleave	Selector and selected-value child probes	()	Done
ScalarFnValidity	Child scalar probe for nullness	()	Done
Union	Type-ID and selected-variant probes	()	Done
Variant	Core-storage validity and scalar probes; optional shredded-value probe	()	Done
FixedSizeList	Parent validity and element probe across the row’s range	()	Done
List	Parent validity, offset and element probes	()	Done
ListView	Parent validity, offset, size, and element probes	()	Done
VarBin	Parent validity and offset probe; slice the byte buffer directly	()	Done
DecimalByteParts	High- and low-part child probes; nullness comes from the high part	()	Done
Map	Entries ListView probe, including its child probes and nullness	()	Done
Patched	Last lane range, optional decoded indices, validity and child probes	PatchedProbeState	Done
PiecewiseSequence	Constant-length arithmetic or bounded checkpoints over piece ends; selected start and multiplier child probes	PiecewiseProbeState	Done
ScalarFn	Stable argument child probes; per-row Constant wrappers passed through VecExecutionArgs	()	Done
ALPRD	Patch index/chunk-offset and left/right child probes; one repeated patch-query result	ALPRDProbeState	Done
BitPacked	Patch and blocked-offset child probes; last validated block range and encoded-offset base	BitPackedProbeState	Done
Delta	Last decoded 1,024-value chunk; current-row parent validity	DeltaProbeState	Done
OnPair	Row-offset and decoded-length child probes; at most one row's code boundaries; existing shared dictionary cache	OnPairProbeState	Done
ParquetVariant	Metadata, raw-value, and typed-value child probes; lazy shredding reconstruction	()	Done
Sparse	One patch search through stable index, chunk-offset, and value child probes; nullness from returned scalar	()	Done
Zstd	Outer validity and a decoded 128-row window after repeated locality; at most 1 MiB of decompressed frame bytes retained	ZstdProbeState	Done
ZstdBuffers	One decoded inner array and its retained probe, capped by 1 MiB of decompressed top-level buffers	ZstdBuffersProbeState	Done

Structural, Map, Patched, and PiecewiseSequence behavior

Structural implementations read children through ProbeState::slot, so repeated parent reads retain the corresponding child probes. List, ListView, and FixedSizeList gather elements directly from the row's range. DecimalByteParts reconstructs the decimal with a bounded stack buffer for lower parts.

Map reads its validated ListView child and wraps the returned entry values in the Map dtype. It preserves null and empty maps without requiring the entry array to execute as a physical Struct array.

Patched retains the last lane's range. Primitive lane offsets and patch indices use direct buffer access. For non-Primitive indices, a repeated visit to the cached lane decodes and retains at most 64 indices; larger lanes continue probing without that decoded cache. One-off reads do not allocate an encoding-owned cache. Patch lookup accounts for the array's slice offset and preserves the parent validity and logical dtype.

PiecewiseSequence reads only the selected start and multiplier through child probes. Constant lengths use direct division; other lengths use a sparse cumulative-end index only for repeated probes. The index has at most 4,096 checkpoints (about 64 KiB of checkpoint entries) regardless of the number of pieces or reads, plus a last-piece hint. A Primitive lengths child is scanned directly; other encodings are decoded in blocks of at most 256 piece lengths during lookup. One-off reads retain no index. Empty pieces, later unused pieces, and deferred range errors preserve the point-read behavior.

ScalarFn reads arguments through stable child slots. Each row's argument values still become length-one Constant arrays and use the existing VecExecutionArgs adapter. It retains no encoding-owned state because the argument values and results belong to the current row. Its computed scalar supplies nullness; a parent validity precheck could recurse through ScalarFnValidity. Both Constant and Canonical function results keep the prior handling.

ALPRD checks its parent validity, searches exception positions through stable child probes, then reads either the patch value or dictionary code and combines it with the low bits. Patches::search_index and the new probe-aware search share the slice and chunk-offset range calculation. For repeated reads of the same patch-bearing row, ALPRDProbeState keeps only the most recent row and its patch hit or miss; it begins retaining after a second consecutive read. The state has constant size, and one-off reads retain nothing.

BitPacked checks parent validity before searching patches. Patch hits return a scalar cast to the parent dtype and do not validate packed bytes. Global-width misses use the existing unpacker; blocked-width misses probe only the selected block's offsets and validate that block before unpacking. BitPackedProbeState retains the last validated block's byte range and the first offset for encoded offset children. Its size is constant. A patch-query cache was tried and removed after a sustained small random-read regression.

Delta checks parent validity on every read. A cache miss decodes only the selected physical 1,024-value chunk through the existing full-chunk algorithm, preserving signed wraparound and deferred child errors. A repeated probe retains that chunk, bounded to at most 8 KiB of numeric values. The cache key combines the caller index with the sliced array's physical offset; nulls do not decode a chunk.

OnPair checks outer validity before accessing children or initializing the dictionary. Primitive row offsets and lengths are read directly; other encodings use stable child slots, allowing their own probe states to persist. The existing shared dictionary cache remains the sole dictionary cache. A repeated probe retains only one row's code boundaries, reusing both on a same-row read or the end boundary for the next row. For at most four compressed codes, child probes beat per-row materialization in the measured short-row fixture; longer runs keep bulk row decoding. Returned strings still own their bytes.

ParquetVariant checks outer validity once and reads metadata, typed values, and raw values through stable child slots. Typed values take precedence; struct typed values merge with raw object fields; a missing typed value falls back to raw bytes. Reconstruction is per-row, so its encoding-owned state remains ().

Sparse searches for a patch once through Patches::search_index_with_probes, then reads a hit through the patch-values slot or clones the fill scalar. Both alternatives already carry nullness, so it avoids a redundant validity-array patch search. Slice offsets and optional chunk offsets stay in the shared Patches search helper. It retains no encoding-owned state.

VarBin directly reads already-hosted Primitive offsets and keeps child probes for other offset encodings. The direct path was added after the final baseline comparison found a 7.3% repeated-read regression in the common Primitive-offset fixture; the final path is 33.5% faster than origin/develop in that fixture.

Zstd checks outer validity and uses the existing rank-aware frame decompressor. One-off reads decode one row. Repeated reads decode one row on the first window miss, then promote a 128-row aligned window only after a second request in it. The state retains one decoded window only when its decompressed frame bytes fit within 1 MiB; an oversized window continues on the one-row path. This preserves random-read performance while reusing frame decompression for localized reads.

ZstdBuffers checks outer validity, then reconstructs and retains one decoded inner array and its RepeatedArrayProbe when the sum of its uncompressed top-level buffers is at most 1 MiB. The inner probe keeps any inner encoding state across reads. Larger wrapped arrays and one-off reads use the existing decode path without a retained inner array.

Tests and benchmarks

• Adds test_scalar_probe_consistency for encodings outside the full compute conformance suite, checking one-off and repeated reads, nullness, revisits, and slices.
• Adds focused coverage for Filter, Interleave, ScalarFnValidity, Union, Variant, FixedSizeList, ListView, and Map, including Map entries stored in a non-Struct encoding.
• Adds Patched tests for nullable rows, patch hits and misses, chunk boundaries, slices, and chunked lane-offset and patch-index children.
• Adds PiecewiseSequence tests for mixed integer widths, empty pieces, non-unit multipliers, overflow, declared-length mismatch, revisits, slices, chunked lengths, and the checkpoint cap.
• Adds ScalarFn probe consistency coverage for nested nullable functions, revisits, slices, and zero-argument functions.
• Adds ALPRD probe coverage for f32/f64 patch hits and misses, nulls, revisits, slices, and manually constructed chunk offsets after two slices.
• Adds BitPacked probe coverage for signed nullable values, patch hits and misses, FoR-encoded offsets, revisits, double slices, chunked patch offsets, and deferred malformed-block errors.
• Adds Delta probe coverage for signed nullable wraparound values, same- and cross-chunk revisits, nested slices, BitPacked delta children, and deferred malformed-chunk errors.
• Adds OnPair probe coverage for nullable Utf8, Binary with compressed child arrays, revisits, nested slices, and null rows under an invalid dictionary.
• Adds ParquetVariant probe coverage for typed and raw rows, outer null, inner Variant null, revisits, and slices.
• Adds Sparse probe coverage for nullable patch values and fill, revisits, and slices.
• Adds Zstd probe coverage for nullable Primitive and Utf8 values, window boundaries, revisits, and slices.
• Adds ZstdBuffers probe coverage for nullable Primitive and Utf8 inner arrays, revisits, and null rows.
• Extends the probe benchmark with Bool, VarBinView, List, Interleave, Map, Patched, PiecewiseSequence, and ScalarFn, alongside the existing Primitive, Dict, and Struct cases. Adds Patched same-lane reads, PiecewiseSequence same-piece reads, and ScalarFn same-row reads, each in one-off and repeated modes.
• Adds alp_rd_probe for f32/f64, with and without exceptions, using random and same-row one-off/repeated reads and explicit patch-hit/miss rows.
• Adds bitpacked_probe for global widths and blocked Primitive/FoR offsets, with and without patches, using random and same-block one-off/repeated reads and explicit patch-hit/miss rows.
• Adds delta_probe for Primitive and BitPacked children and nullable signed values, comparing random, same-chunk, and two-chunk alternating reads in one-off and repeated modes.
• Adds OnPair probe for short and long strings, Primitive and recursively compressed children, and random, sequential, and same-row one-off/repeated reads.
• Adds ParquetVariant probe for typed, raw, and mixed rows under random and same-row one-off/repeated reads.
• Adds Sparse probe for non-null and nullable values under random, same-patch, and same-fill one-off/repeated reads.
• Adds Zstd zstd_probe for Primitive, nullable Primitive, and Utf8 values under random, same-frame, and same-row one-off/repeated reads.
• Adds zstd_buffers_probe for Primitive, nullable Primitive, and Utf8 inner arrays under random and same-row one-off/repeated reads.
• Extends the core probe benchmark to cover Constant, Decimal, Null, Filter, FixedSizeList, ListView, Union, Variant, VarBin, and the internal ScalarFnValidity array. Adds dedicated probe benchmarks for ByteBool, Sequence, and DecimalByteParts.

Verification

• cargo nextest run -p vortex-array -p vortex-bytebool -p vortex-sequence: 4,201 passed, 1 skipped.
• cargo nextest run -p vortex-python: 35 passed.
• Bool and VarBinView were benchmarked before and after with both one-off and repeated reads. VarBinView medians were stable or lower; Bool measurements varied between runs, with no sustained regression established.

API Changes

Uses the probe API already introduced in #9843. The migrated implementations retain scalar_at as a compatibility shim. Making probe_scalar required, removing OperationsVTable::scalar_at, and changing the public scalar/validity accessors remain pending under #9971. PythonArray scalar access remains unsupported.

AI Usage

• GPT-6-Sol implemented the changes with Codex. The human reviewed all the code and can explain every line of it. Benchmark results and the origin/develop comparisons are done by Codex.

Final point-read comparison to baseline

All the results below are obtained only on a development machine (macOS with Apple Silicon). Not representative of production performance.

The baseline is the implementation at origin/develop commit d270200d65; The benchmark source files, benchmark-only Cargo manifests, and Cargo.lock were copied from the migrated branch into that source tree, so both sides used the same fixtures and dependency versions. Each command used cargo bench -p <crate> --bench <name> -- --min-time 0.1 --sample-count 15 on the same machine with Rust 1.99.1. Values are Divan medians in microseconds per iteration. Each iteration uses one retained probe for the indicated number of scalar reads.

Array	Fixture	Reads	Baseline (µs)	Migrated (µs)	Change
Bool	nullable random	256	4.957	4.624	-6.7%
Constant	fixed scalar, random	256	2.666	2.103	-21.1%
Decimal	decimal128, random	256	4.541	3.957	-12.9%
Null	all null, random	256	1.958	1.260	-35.6%
VarBinView	nullable strings, random	256	6.290	5.707	-9.3%
ByteBool	nullable random	256	14.080	4.957	-64.8%
Sequence	i64 arithmetic, random	256	8.582	2.333	-72.8%
Filter	nullable input, random	256	65.040	55.490	-14.7%
Interleave	two nullable inputs, random	256	34.240	19.450	-43.2%
ScalarFnValidity	nullable binary function, random	256	31.830	24.700	-22.4%
Union	two variants, random	256	23.490	20.040	-14.7%
Variant	constant core storage, random	256	7.916	8.458	+6.8%
FixedSizeList	nullable three-element lists, random	256	105.100	35.160	-66.5%
List	nullable three-element lists, random	256	108.800	45.580	-58.1%
ListView	nullable three-element views, random	256	115.100	48.490	-57.9%
VarBin	nullable Primitive offsets, random	256	14.290	9.499	-33.5%
DecimalByteParts	three lower parts, random	256	26.700	26.200	-1.9%
Map	nullable entries, random	256	160.900	62.580	-61.1%
Patched	Primitive patch indices, random	256	25.700	8.332	-67.6%
PiecewiseSequence	variable piece lengths, random	256	707.100	29.950	-95.8%
ScalarFn	nullable binary function, random	256	131.700	119.100	-9.6%
ALPRD	f32 with patches, random	256	144.400	118.900	-17.7%
BitPacked	blocked FoR with patches, random	256	93.770	52.870	-43.6%
Delta	Primitive u32 deltas, random	256	299.300	237.600	-20.6%
OnPair	short strings, compressed children, random	256	952.100	203.600	-78.6%
ParquetVariant	mixed typed/raw, random	256	36.540	30.450	-16.7%
Sparse	nullable patches and fill, random	256	13.160	7.416	-43.6%
Zstd	Primitive, random	64	337.200	337.500	+0.1%
ZstdBuffers	Primitive inner array, random	64	573.100	9.957	-98.3%
PythonVTable	Scalar probing is unsupported by this vtable	—	N/A	N/A	N/A

Localized repeated reads show the benefit of each bounded cache. This table uses 256 reads per iteration except the two Zstd rows, which use 64.

Array	Fixture	Baseline (µs)	Migrated (µs)	Change
Patched	same lane	45.700	10.120	-77.9%
PiecewiseSequence	same piece	639.700	21.370	-96.7%
ALPRD	same patched row	143.700	20.790	-85.5%
BitPacked	same FoR block	38.870	4.374	-88.7%
Delta	same chunk	292.100	3.874	-98.7%
OnPair	sequential short strings	973.900	70.580	-92.8%
Sparse	same patch row	32.240	14.620	-54.7%
Zstd	same frame	341.700	11.700	-96.6%
ZstdBuffers	Utf8 inner array, random	4770.000	73.990	-98.4%

The VarBin row uses the final Primitive-offset fast path from cf5a9da2aa; its first forwarding implementation measured 15.33 µs repeated and 17.49 µs one-off, versus 14.29 µs and 16.66 µs on the baseline. Direct host offset reads reduced the final repeated median to 9.499 µs and the one-off median to 10.74 µs. The Variant constant-core fixture remained 6.8% slower for repeated reads; its one-off median improved from 7.332 to 6.541 µs. A trial of parent validity through ProbeState::is_valid did not improve Variant repeated reads and was reverted. Zstd random reads are unchanged in the Primitive fixture, while same-frame reads benefit from its bounded window. These are microbenchmarks over the listed fixtures; the per-milestone sections above give the broader access-pattern results.

Benchmark commands compiled and completed for both source trees. Focused tests written for the final four encodings and Rust lint/format checks were not run under the repository’s user-chosen verification policy. Markdown documentation stays outside the commits.

[codspeed]

Merging this PR will degrade performance by 19.67%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

• bench_compare_primitive[(10000, 2)]

⚠️ 18 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• chunked_dict_primitive_into_canonical[f64, (1000, 1000, 10)]
• bench_compare_sliced_dict_primitive[(3333, 10000)]
• chunked_dict_primitive_canonical_into[f64, (1000, 1000, 10)]
• optimize_lookup_predicate[ids=256, shape=in_list]
• random_i128[0.01]
• bench_dict_mask[(0.9, 0.9)]
• chunked_dict_primitive_into_canonical[u64, (1000, 1000, 10)]
• multipolygon_mixed_validity
• filter_run_end[len=4096_run=16_density=0.1]
• decompress[u32, (1000, 256)]
• …and 8 more

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 3 regressed benchmarks
✅ 2215 untouched benchmarks
🆕 8 new benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	density_sweep_dense_runs[0.001]	28.5 µs	47 µs	-39.26%
❌	Simulation	filter_powerlaw_by_mostly_true[250000]	98.1 µs	145.1 µs	-32.42%
❌	WallTime	filter_avx2[64]	2.1 µs	2.3 µs	-10.83%
⚡	WallTime	starts_with_avx512[8, VarBin]	46.3 µs	40.7 µs	+13.76%
🆕	Simulation	once[bool]	N/A	185.2 µs	N/A
🆕	Simulation	once[interleave]	N/A	520 µs	N/A
🆕	Simulation	once[list]	N/A	1.2 ms	N/A
🆕	Simulation	once[varbinview]	N/A	196.2 µs	N/A
🆕	Simulation	repeated[bool]	N/A	156.1 µs	N/A
🆕	Simulation	repeated[interleave]	N/A	547.4 µs	N/A
🆕	Simulation	repeated[list]	N/A	1.1 ms	N/A
🆕	Simulation	repeated[varbinview]	N/A	172.9 µs	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing jiengup:migrate-every-encoding (7477dcb) with develop (ed119bf)}