[connortsui20]

Research notes on the Vortex type system: extension types, the Decimal rework, wide integers, FixedSizeBinary as a core type, and casts. This PR exists so the notes can be read and discussed on GitHub. It is not meant to merge.

For review, start with research/decisions.md. It records the settled decisions and the open questions, each with a recommendation, and the other notes follow it. Each topic then has a short summary (research/extension-types/, research/decimal/) and an agents/ directory with the detailed report, file and line references against develop at 5fc5cb8, and reproductions.

The main decisions so far:

• Extension types are opaque by default. A type implements its behavior as rewrite rules on arrays, registered in the session with the type, and a rewrite rule lifts generic structural arrays such as Constant and Dict out from under the extension wrapper.
• ExtensionArray::try_new checks values, and a new new_unchecked serves kernels.
• Decimal stays a built-in leaf with one buffer. The precision fixes the buffer width, so each decimal dtype has exactly one canonical encoding, and a Narrow encoding stores a column at a lower precision with the same scale.
• Vortex adds I128, I256, U128 and U256, and FixedSizeBinary becomes a core type.
• A cast exists only where a rule for that pair of dtypes exists.

The notes also describe how the open Narrow PRs (#10223, #10332, #10333) and #10230 fit this plan.

🤖 Generated with Claude Code

[codspeed]

Merging this PR will regress 5 benchmarks

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 2 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 19 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 4 improved benchmarks
❌ 5 regressed benchmarks
✅ 2430 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	test_scan_filter_to_arrow_array[int_float]	3.5 ms	4 ms	-12.97%
❌	CPU Simulation	test_scan_filter[int]	3.1 ms	3.5 ms	-11.28%
❌	CPU Simulation	test_repeated_scan_filter[int_float]	3.4 ms	3.8 ms	-10.88%
❌	CPU Simulation	test_repeated_scan_filter[int]	2.4 ms	2.7 ms	-10.53%
❌	CPU Simulation	test_scan_filter_to_arrow_array[int]	2.5 ms	2.8 ms	-10.27%
⚡	CPU Simulation	filter_powerlaw_by_random[250000]	174.3 µs	131.3 µs	+32.74%
⚡	Walltime	mul_u16_nonnull_avx512	13.7 µs	11.6 µs	+18.12%
⚡	Walltime	mul_u32_nonnull_avx512	13.9 µs	12.1 µs	+15.03%
⚡	Walltime	subtract_shapes_avx512[(32768, PerRowPerRow)]	14 µs	12.2 µs	+14.51%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/ext-types-research (372e76a) with develop (423354d)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩