Tracking Issue: Fix Extension Types #11114

This is a tracking issue for making Vortex's extension types sound and consistent.

• Subissue of Epic: Harden Type System #10730 and Epic: Extension Types #7683.

Design

Arguably the most important feature of Vortex is its extensibility. We allow extension points in many places, including the type system. However, logical extension types have always been somewhat unwieldy, where we constantly find ourselves running into edges and corners. This is mostly because our understanding of what an extension type should be is generally different from what we have actually implemented, and that causes bugs and other issues.

So this issue is a place where we can more formally define the design and ideas behind extension types, and we can gradually make changes to get closer to our idealized vision of extension types.

Extension types are opaque

This is probably the most apparent change that we need to make. When we wrap an extension type around a storage type, the extension type should not inherit any behavior from the storage type unless it explicitly opts in. The only way to get to the storage is to ask for it with ext_storage.

In Rust, a close example to this is a newtype, where we wrap an existing type with a tuple struct with a single argument. Unless we mark that argument as pub, this type is fully opaque. If we want to forward behavior to the inner type (like fields, methods, and trait implementations), we must manually implement that logic (it does not happen for free).

However, doing this for every shared behavior is somewhat cumbersome and tedious, so ideally we want to have ways to conveniently forward logic. See #11115 for more details on a proposal about this.

Top-level nullability

An extension type can't choose its top-level nullability: it must accept both a nullable and a non-nullable storage dtype, and its validator can't depend on which one it gets. #11012 made this change. Note that validators can still constrain nested nullability, since that belongs to the type (see the next section).

This is subtle because the validator sees the whole storage dtype, top-level nullability included, so it looks like a type should be able to reject one of them. But Vortex changes the top-level nullability of arrays all the time without changing any values: take with nullable indices and mask both make an array nullable, and min_max builds non-nullable result dtypes. Before #11012, each of these validated the type again and panicked if the type rejected the new nullability.

A top-level null means the whole value is missing, which says nothing about what the type means. Whether a column can have missing values is a property of the column, the same as for every built-in dtype. Nested nulls are different, because they're part of a value.

The storage declaration in the next section turns this from a documented rule into a structural one: storage_dtype takes no nullability at all, and ExtDType::try_new(metadata, nullability) applies the top-level nullability to the declared storage dtype itself, so a type can't depend on it even by accident.

Extension metadata rework

Right now, an extension type is handed a storage dtype and decides whether to accept it (ExtVTable::validate_dtype). We want to change this so that each extension type declares its storage dtype from its metadata, the same way an encoding declares the dtypes of its children.

This is how array deserialization already works in Vortex, but the more important reason is related to nested nullability in extension storage types. We have decided that an extension type over nested storage must choose exactly which of its inner levels can be null, because that's part of what the type means.

For example, a LineString is a list of non-null coordinates, and the elements of a Vector are never null. On the other hand, a range type like Postgres' int8range, stored as a struct of lower and upper bounds, must declare both bounds as nullable, since a null bound means the range is unbounded on that side. [5, inf) is a real range, and declaring the bounds non-nullable would make it impossible to represent. A range with two null bounds (the range that contains everything) is still a valid, non-null value, which is different from a null range.

This means that we are constraining extension types to only allow 1 kind of nullability per level in nested storage. Please let us know if this breaks your use case.

Here is an updated (simplified) vtable:

pub trait ExtVTable {
    fn storage_dtype(metadata: &Self::Metadata) -> DType;
    fn validate_metadata(metadata: &Self::Metadata) -> VortexResult<()> { Ok(()) }
    fn deserialize_metadata(&self, bytes: &[u8], stored_storage: &DType) -> VortexResult<Self::Metadata>;
}

This implies that metadata holds every fact that decides the storage. Some of those facts only live in the storage dtype today, like the element type and dimension of Vector, the element type of FixedShapeTensor, and the dimension of Point and Rect, so they will need to move into the in-memory metadata.

Note that none of this changes the file format, since the serialized metadata stays the same. It does change the ExtVTable API, and reads will reject a file whose stored storage dtype doesn't match the declaration, so a file in any other form will need a read plugin.

Steps

• Fix the bugs that give wrong results or panics today, so that every encoding and read path gives the same answer
• Declare the storage dtype from the metadata, and read only the declared storage
• Reject every semantic operation that no declaration or type rule covers, with hardcoded opt-ins for the extension types in the repository
• Let each extension type own its casts and other rules
• Keep the extension type at every boundary (Arrow, DataFusion, DuckDB, Python, JNI)
• Check extension values when arrays are built or imported
• Document the contract for extension type authors

The first two steps can happen in parallel, and the third needs only the first. #11115 later replaces the hardcoded opt-ins with claims. Most of the later steps build on the third, but the value checks and most of the boundary work only need the storage declaration. Within the steps, a few PRs have to land in order, because each one alone breaks something that works today. The Arrow export has to land before casts stop reinterpreting storage, since callers that ask Arrow for a storage-like type (such as Int64 for a timestamp column) would otherwise lose their only lossless path. The spatial nullability fix has to land before the storage declaration, which makes reads strict. A check on 2026-11-03 found that no Vortex producer wrote a spatial dtype with a nullable nested level. Only storage built by hand through the public Rust API can have one, so such files stop reading, and no read plugin is needed.

Unresolved questions

• Open question: which generic structural encodings (Constant, Dict, RunEnd, Sparse, Chunked, Masked, Interleave) should sit above the extension wrapper, and should a canonical extension array always wrap canonical storage? See Keep generic structural encodings above other encodings #10748.
• Open question: what should PartialEq, PartialOrd, and a future Hash on Scalar mean for extension values? Expression results won't depend on them, but Rust code like maps of constants and tests still does.

Implementation history

• feat(aggregate): add ExtensionStoragePushdown to push an aggregate into an extension type's storage #10566 Add ExtensionStoragePushdown to run an aggregate on the storage of an extension type (2026-10-20)
• feat(aggregate): register is_sorted and min_max for temporal and UUID extension types #10616 Register is_sorted and min_max storage pushdown for the temporal types and Uuid (2026-10-20)
• Return errors for invalid temporal types and values #11015 Return errors for invalid temporal types and values (2026-10-30)
• Add execution path consistency assertions #11013 Add execution path consistency assertions (2026-10-30)
• Change extension dtype nullability without validating again #11012 Change extension dtype nullability without validating again (2026-11-02)
• Require matching dtypes for extension comparisons #11016 Require matching dtypes for extension comparisons (2026-11-02)
• Preserve extension literals in DataFusion comparisons #11067 Preserve extension literals in DataFusion comparisons (2026-11-02)
• Validate extension storage and return scalar read errors #11014 Validate extension storage and return scalar read errors (2026-11-02)
• Document temporal storage and binary comparison execution #11099 Document temporal storage and binary comparison execution (2026-11-02)
• Reduce constant projections through select and extension storage #11056 Reduce constant projections through select and extension storage (2026-11-03)
• fix(array): accept timestamps beyond jiff's range #10953 Accept every i64 timestamp, including values beyond jiff's range, and widen DateTimeParts days that don't fit in i32 (2026-11-03)