[connortsui20]

Summary

• Tracking Issue: Tracking Issue: Add Widen encoding for integers and decimals #10733

Internal integer children now keep their signedness when the compressor narrows them. Each call to the deprecated encode_values marks a child whose dtype policy still needs a decision.

Changes

• Replace all 24 narrowing calls across 17 files with WidenArray::encode_values, using the same TODO(connor) reason at each call.
• Preserve signed List offsets, signed ListView offsets and sizes, FSST lengths and code offsets, and DateTimeParts parts. Non-negative signed values from 128 to 255 now need i16 rather than u8. Unsigned children keep their behavior.
• Port the boundary, nullability, and empty-input cases to the existing encode_values tests. Add compact list and string serialization round trips that exclude vortex.widen. The list case asserts signed offsets and fails with the old narrowing.
• Regenerate five golden snapshots. Only signed dtype and scalar labels change. Every byte count stays the same.

API Changes

PrimitiveArrayExt::narrow is removed. Newly written files retain signed dtypes for signed internal children. Encoding IDs and metadata formats do not change.

Source audit and signed-child readers

The brief's table contains 24 calls, rather than 25. All 24 calls are migrated. The binary VarBin scheme builds u64 offsets, and OnPair produces unsigned children, so those sites do not change signedness. Dictionary codes, RunEnd ends, RLE indices and offsets, and patch and sparse indices are also unsigned. List and ListView callers can provide signed children. FSST produces i32 lengths and i32 or i64 offsets. DateTimeParts produces signed parts.

All locations below are relative to the PR source tree. CUDA results are static inspection only.

Child	Reader	Location	Result
List offsets	Validation and offset access	vortex-array/src/arrays/list/array.rs:213, :231, :308	Accepts integers and dispatches signed types.
ListView offsets and sizes	Validation, element mask, and rebuilding	vortex-array/src/arrays/listview/array.rs:239, :269, :435; vortex-array/src/arrays/listview/rebuild.rs:165	Reinterprets non-negative values to unsigned at the same width before unsigned dispatch.
VarBin offsets, including FSST code offsets	Validation and canonicalization	vortex-array/src/arrays/varbin/array.rs:238; vortex-array/src/arrays/varbin/vtable/canonical.rs:29	Accepts integers and dispatches all integer types.
FSST lengths and code offsets	Decode and builder append	encodings/fsst/src/canonical.rs:34, :70, :98; encodings/fsst/src/array.rs:362	Uses all integer types.
DateTimeParts days, seconds, subseconds	Decode	encodings/datetime-parts/src/canonical.rs:69, :135	Dispatches days over all integer types and casts the other parts to i32 before typed reads.
OnPair children	Decode	encodings/onpair/src/canonical.rs:42, :73; encodings/onpair/src/decode.rs:51, :66	Accepts integer children and widens to decoder types. Current producers are unsigned.
List and ListView offsets and sizes	Arrow export	vortex-arrow/src/executor/list.rs:97, :155, :162	Casts to Arrow's required offset type.
VarBin offsets	Arrow export	vortex-arrow/src/executor/byte.rs:108	Casts to Arrow's required offset type.
List and ListView offsets and sizes	DuckDB export	vortex-duckdb/src/exporter/list.rs:91; vortex-duckdb/src/exporter/list_view.rs:129	Dispatches all integer types.
VarBin offsets	DuckDB canonical export	vortex-duckdb/src/exporter/canonical.rs:26, :31	Canonicalizes to VarBinView through the all-integer CPU decode.
List offsets and ListView offsets and sizes	CUDA dispatch	vortex-cuda/src/kernel/arrays/list.rs:88; vortex-cuda/kernels/src/list.cu:36; vortex-cuda/kernels/src/list_view.cu:227	Integer dispatch and kernels include signed widths.
FSST code offsets	CUDA decode	vortex-cuda/src/kernel/encodings/fsst.rs:158, :202; vortex-cuda/kernels/src/fsst.cu:298	Dispatches by unsigned width and reads the identical representation of non-negative signed offsets.
FSST and OnPair lengths	CUDA Arrow offsets	vortex-cuda/kernels/src/arrow_offsets.cu:56	Kernels include all eight integer types.
OnPair code offsets	CUDA window boundaries	vortex-cuda/kernels/src/onpair.cu:107	Kernels include all signed and unsigned offset widths. Token codes remain unsigned.

Snapshots

Snapshot	Change	Root bytes, unchanged
compact/list_of_int_runs	Offsets u16 to i16.	4,434
regular/list_of_int_runs	Offsets and their patch values u16 to i16.	11,146
regular/string_fsst_structured	Lengths and scalar labels u8 to i8, code offsets u32 to i32.	151,313
regular/string_low_cardinality	Dictionary values' FSST lengths and offsets u8 to i8.	8,373
regular/struct_mixed	Category dictionary values' FSST lengths and offsets u8 to i8.	57,525

Every stored range fits the signed type at the existing width. No byte-count changes need explanation.

Checks and compatibility

• Scoped clippy for vortex-array, vortex-btrblocks, vortex-compressor, and vortex-fastlanes, with all targets, all features, and warnings denied: passed.
• Formatting with nightly-2026-09-10 for those four crates: passed.
• cargo nextest run -p vortex-array: 5,189 passed, one skipped.
• cargo nextest run -p vortex-btrblocks --all-features: 92 passed.
• cargo nextest run -p vortex-compressor -p vortex-fastlanes: 1,240 passed.
• cargo nextest run -p vortex-fsst -p vortex-onpair -p vortex-datetime-parts: 242 passed, one skipped.
• cargo nextest run -p vortex-file: 189 passed. No file-test changes were needed.
• All three golden variants passed during snapshot regeneration.
• Before migration, the new list regression failed with actual U16 and expected I16. After migration, both cases passed.
• No .narrow( calls remain. All 24 migrated calls have the exact deprecation reason. git diff --check passed.
• Commit 1 passed scoped clippy and all 5,209 array tests before the call sites changed. Commit hooks passed for both commits without bypasses. The pre-push hook also passed workspace-wide clippy with all targets, all features, and warnings denied.
• Old-reader check: the directly built 0.90.0 reader passed 36 fixtures, failed none, and skipped the same seven newer decimal fixtures for both base 9bf403a1ce and WD6. The per-fixture results match exactly in subset mode. All 43 fixtures were generated on each revision. The ClickBench download succeeded, and both runs reused the same deterministic sample. Both temporary compatibility worktrees were removed.
• Coverage gap: compressed fixtures are TPC-H lineitem, TPC-H orders, and ClickBench, with no list fields. Synthetic List and ListView fixtures use flat layouts. No new fixture was added. The new integration test covers compressed lists with the current reader.
• CUDA cannot be built locally. The PR path filter skipped the dedicated CUDA workflow, so a manual run was started on exact head b2bd758df8. Its results are pending.

Performance evidence and remaining CI work

Local btrblocks comparisons were skipped at the maintainer's request. CodSpeed does not run the btrblocks benchmarks.

The valid WD3 report compares 7dd6ba4 with 2fb259b. All 12 take_map cases are classified NoChange, with latency differences from -1.03% to +0.96%. The valid WD4 report compares 419b196 with 37031b1. All 12 cases are NoChange, with latency differences from -0.31% to +6.34%. WD4 carries a runner-difference warning. Neither report has an unexpected-base condition. WD1's comment uses head 75c7f4a, while its final head is a56a48e, so it is excluded as evidence for the final PR.

The take_map fixture uses u64 indices and values directly and calls no narrowing code. The selected hashbrown 0.17.1 default builder uses foldhash 0.2.0 random state, with per-hasher and shared seeds. A fixed fixture RNG does not fix the hash layout. This establishes a possible source of variance, not the cause of the old regressions.

WD6's own CodSpeed comparison, any needed flame graph analysis, and develop-history comparison remain pending the PR benchmark run. Only reports with the correct base and head will be used. No take_map code changed.

[codspeed]

Merging this PR will regress 1 benchmark

3 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 4 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

⚡ 11 improved benchmarks
❌ 1 regressed benchmark
✅ 2258 untouched benchmarks
⏩ 370 skipped benchmarks¹
🗄️ 30 archived benchmarks run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	filter_powerlaw_by_random[10000]	531.1 ns	891.9 ns	-40.45%
⚡	CPU Simulation	encode_primitives[u8, (1000, 4)]	112.8 µs	74 µs	+52.32%
⚡	CPU Simulation	encode_primitives[u8, (1000, 2)]	112.6 µs	74 µs	+52.17%
⚡	CPU Simulation	encode_primitives[u8, (1000, 8)]	113.5 µs	74.9 µs	+51.44%
⚡	CPU Simulation	encode_primitives[u8, (1000, 32)]	115 µs	76.7 µs	+50%
⚡	CPU Simulation	encode_primitives[u8, (1000, 512)]	130.5 µs	92.4 µs	+41.27%
⚡	CPU Simulation	encode_primitives[u8, (2000, 2)]	164.2 µs	125.6 µs	+30.7%
⚡	CPU Simulation	encode_primitives[u8, (2000, 8)]	165.2 µs	126.6 µs	+30.49%
⚡	CPU Simulation	encode_primitives[u8, (2000, 32)]	167.1 µs	128.4 µs	+30.08%
⚡	CPU Simulation	encode_primitives[u8, (2000, 512)]	182.9 µs	144.4 µs	+26.65%
⚡	CPU Simulation	test_repeated_scan_scalar_at[int_str_float]	497.7 µs	415.5 µs	+19.78%
⚡	Walltime	starts_with_avx2[8, VarBin]	46.6 µs	40 µs	+16.56%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/widen-encode-values (b2bd758) with develop (10a4d3a)}

Footnotes

1. 370 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. 30 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩