ReadRel.LocalFiles cannot express path-derived partition fields: motivation and cross-engine survey

[nielspardon]

The gap

ReadRel.LocalFiles cannot express a partitioned read. Partitioned datasets commonly encode some field values in the file path (.../year=2024/month=03/...) or in table metadata rather than in the file contents, and a consumer needs those values to reconstruct the full records. Today a Substrait plan can name the files and the schema, but there is nowhere to say "field 3 of this item has the constant value 2024, and is not stored in the file."

This is not a theoretical gap. Three shipped Substrait implementations lose data because of it:

1. datafusion-substrait — the physical producer drops the values. physical_plan/producer.rs builds each FileOrFiles from partition_index, length, path_type and file_format only. The PartitionedFile.partition_values it reads from are discarded.
2. datafusion-substrait — the physical consumer cannot restore them. physical_plan/consumer.rs uses PartitionedFile::new_from_meta(...), which leaves partition_values empty. So a DataFusion → Substrait → DataFusion round trip of a Hive-partitioned Parquet scan returns rows missing their partition columns.
3. substrait-java (Spark module) hardcodes "not partitioned." ToLogicalPlan.scala calls createHadoopFsRelation(spark, location, new StructType(), schema, ...) — that empty StructType() is the partition schema.
4. Acero builds a flat vector<fs::FileInfo> with a default (empty) partitioning from local_files (relation_internal.cc), even though Arrow's dataset layer fully supports partition expressions. The gap is in the Substrait mapping, not in Arrow.

And for the lakehouse formats the values are not recoverable from the file path at all: Iceberg data-file locations are arbitrary and the manifest's partition struct is authoritative (format/spec.md: "The partition struct stores the tuple of partition values for each file"), and Delta's transaction log is authoritative rather than the path. A consumer handed only a file list cannot reconstruct them by parsing. So this is a correctness requirement, not a convenience.

PR #1139 proposes a fix. On the 2026-08-12 community sync the PMCs asked how universal the concept is across engines and how other systems represent it, so this issue records the survey behind that question.

---

Survey method

For each system I read the actual type or message that carries a per-file partition value — not the documentation — and recorded four things:

1. Does the concept exist at all?
2. Is the value carried in the plan/split, or re-derived from the file path at read time?
3. Is the value typed, or a string parsed later?
4. How is the target field identified — ordinal position, name, or field id?

Scope is value-based (identity) partitioning only. Hash, range and bucket partitioning are a different feature and are excluded everywhere below.

---

Summary of findings

The concept is universal: 20/20 systems surveyed have it. Not one engine that reads file-based tables lacks a notion of "this field's value is constant for this file and is not stored in the file."

The representation is not converged — but the divergence is predictable from the carrier, and that turns out to be the useful part of the answer:

Axis	Split	What predicts the choice
Carried in the plan/split vs re-derived from path	13 carry / 7 re-derive	Systems that separate planning from execution across a process boundary all carry it. Single-process readers re-parse the path.
Typed value vs string	12 typed / 8 string	The 8 string-based ones all inherit the Hive metastore convention (list<string> values) or serialize through a text/JSON metadata format. Every system whose carrier is a typed IR carries typed values.
Field identified by	8 ordinal / 11 name / 1 field-id	Ordinal where the carrier is a positional row or list; name where the carrier is a map.

The first row is the one that matters for Substrait: Substrait is a plan interchange across a process boundary, and every peer in that class carries partition values explicitly — Impala's Thrift descriptors, Comet's protobuf, DataFusion's protobuf, Dremio's protobuf, and Gluten's Substrait fork. This argument does not depend on Gluten.

---

The survey

System	Where the per-file partition value lives	Carried or re-derived	Value	Field id'd by
Apache Spark	PartitionedFile.partitionValues: InternalRow (FileScanRDD.scala)	carried	typed (against partitionSchema)	ordinal
Apache DataFusion	PartitionedFile.partition_values: Vec<ScalarValue> (datasource/src/mod.rs)	carried	typed	ordinal
DataFusion protobuf	PartitionedFile.partition_values = repeated ScalarValue (datafusion.proto)	carried	typed	ordinal
Apache DataFusion Comet	SparkPartitionedFile.partition_values = repeated spark_expression.Expr (literals) (operator.proto)	carried	typed	ordinal
Apache Impala	THdfsPartition.partitionKeyExprs: list<Exprs.TExpr> — "These are Literal expressions" (CatalogObjects.thrift)	carried	typed	ordinal
Dremio	PartitionValue { type; column; bit/int/long/float/double/binary/string_value } (partition.proto)	carried	typed	name
Apache Gluten (Substrait fork)	partitionColumn { string key; string value; }, repeated ... partition_columns = 17 (vendored algebra.proto)	carried	string	name
Velox	HiveConnectorSplit.partitionKeys: unordered_map<string, optional<string>> (HiveConnectorSplit.h)	carried	string	name
Trino	HiveSplit.partitionKeys: List<HivePartitionKey>, record HivePartitionKey(String name, String value) (HivePartitionKey.java)	carried	string	name
Presto	HivePartitionKey(String name, Optional<String> value) (HivePartitionKey.java)	carried	string	name
DuckDB	MultiFileConstantEntry { MultiFileGlobalIndex column_idx; Value value; } (multi_file_data.hpp)	re-derived, then carried in bind data	typed	ordinal
Apache Arrow Datasets / Acero	Fragment::partition_expression(): compute::Expression — a conjunction of field == literal (dataset.h)	re-derived (Partitioning::Parse(path))	typed	name
ClickHouse	HivePartitioningKeysAndValues = Map<string_view, string_view> parsed from the path (HivePartitioningUtils.h)	re-derived	string → cast	name
Polars	HivePartitionsDf(DataFrame) — materialized typed values (plans/hive.rs)	re-derived	typed	name
Apache Flink	PartitionPathUtils.extractPartitionSpecFromPath → LinkedHashMap<String,String>, then PartitionFieldExtractor casts per LogicalType	re-derived	string → cast	name
Apache Hive (metastore)	Partition.values: list<string> — "// string value is converted to appropriate partition key type" (hive_metastore.thrift)	carried	string	ordinal
Apache Iceberg	ContentFile.partition(): StructLike, typed per the spec's partition struct (spec.md); readers get Map<Integer fieldId, ?> from PartitionUtil.constantsMap	carried (path cannot be parsed)	typed	field id
Delta Lake	add.partitionValues: Map[String,String] plus a normative Partition Value Serialization section (PROTOCOL.md)	carried	string	name
Apache Paimon	DataSplit.partition: BinaryRow	carried	typed	ordinal
Apache Hudi	PartitionPathParser.getPartitionValues(...) → Object[], typed against the writer schema	re-derived	typed	ordinal
BigQuery (external tables)	detection modes AUTO / STRINGS / CUSTOM; CUSTOM declares types inline: gs://b/t/{dt:DATE}/{val:STRING}	re-derived	typed (or all-string by choice)	name

(DataFusion's Rust and protobuf representations are counted as one system in the §"Summary" counts, giving 20.)

---

Four representational families

(a) Field reference + typed literal, per file — DuckDB (column_idx + Value), DataFusion (positional ScalarValue), Comet (repeated Expr literals), Impala (list<TExpr> literals), Dremio (column + typed oneof), Spark (InternalRow against partitionSchema), Paimon (BinaryRow). This is the family PR #1139 proposes.

(b) Name → string, per file, cast at read time — Gluten's current fork, Velox, Trino, Presto, Delta, ClickHouse, Flink, the Hive metastore. All of these trace back to the Hive metastore convention.

(c) A boolean expression that holds over the file — Arrow Datasets / Acero: partition_expression is and(equal(field_ref("year"), literal(2024)), ...). Equivalent information, also typed. Notable because Acero is a Substrait consumer, and this shape maps onto (field, value) pairs essentially one-to-one.

(d) Field id → typed value — Iceberg, keyed by schema field id rather than position or name, consistent with its general approach to schema evolution.

---

Why typed literals rather than strings

Every string-based representation in the survey has had to bolt on a normative serialization spec, a null sentinel, or both — and they still disagree with each other:

• Strings force a normative per-type serialization spec. Delta has an entire Partition Value Serialization section in PROTOCOL.md, and it still admits ambiguity: "A timestamp value in a partition value may be stored in one of the following ways" (local time, or UTC ISO8601). Substrait would have to write and police the same table.
• Strings need a null sentinel, and the sentinel collides with real values. Delta: "An empty string for any type translates to a null partition value" — so an empty-string partition value is unrepresentable. Hive and Gluten use __HIVE_DEFAULT_PARTITION__; Trino rewrites it to \N; Velox and Arrow use optional<string>. Four different answers to a problem Expression.Literal does not have.
• String → value parsing is session-dependent. ClickHouse's own comment: "Hive partition values are essentially a string-to-type cast, so we honour cast_string_to_date_time_mode (rather than date_time_input_format)". A plan whose meaning depends on the consumer's cast settings is not a portable plan.
• Type inference from path text is fragile, and engines have retreated from it. BigQuery added a CUSTOM mode so you can declare {dt:DATE} instead of relying on AUTO ("The values must validly parse as the specified type or the query fails"). DuckDB added hive_types and hive_types_autocast. Hudi's path parser contains "if the field is not present in the schema, we assume it is a string" and "Only one date field based partition is supported".
• Typed values are what optimizers need. DataFusion derives exact statistics from partition values — min = max = partition_value, null_count = 0, distinct_count = 1 — expressly "to use partition column bounds for pruning and planning". Range pruning against a string encoding of a date does not work.
• Gluten's own code shows the round-trip cost. Its producer formats Spark's already typed InternalRow partition values into strings — VeloxIteratorApi.scala uses a DateFormatter, a UTC TimestampFormatter, Decimal.toPlainString and decodes binary through new String(bytes, UTF_8) (lossy for non-UTF-8 bytes) — and the ClickHouse consumer parses them back with buildFieldFromString (FileReader.cpp), matching column names case-insensitively via boost::to_lower_copy. Typed values in the plan delete that entire layer.

The sharpest single data point: Gluten and Comet solve the identical problem — a Spark JVM planner shipping a per-file scan description over protobuf to a native executor. Gluten chose {string key, string value}. Comet chose repeated Expr typed literals and converts them straight into DataFusion ScalarValues (planner.rs). PR #1139 proposes the Comet answer, not the Gluten one.

---

How consumers would implement it

Consumer	Maps partition_fields to
DataFusion	PartitionedFile.partition_values (ScalarValue) — direct
DuckDB	MultiFileConstantEntry { column_idx, Value } — direct, same shape
Acero	Fragment::partition_expression as and(equal(field_ref, literal), ...) — direct
Spark (substrait-java)	PartitionedFile.partitionValues plus a real partitionSchema
Velox / Trino	format the literal into their string partitionKeys map (lossy direction, but they already do this internally)

---

Design points the survey speaks to

Per-item pairs vs. one scan-level partition schema plus positional values. PR #1139 attaches (field, value) pairs to each item, which repeats the field reference per file. The more compact alternative — declare the partition columns once on LocalFiles and give each item a positional value list — is what Spark, Comet and DataFusion do (FileScanExecConf.table_partition_cols + PartitionedFile.partition_values; Comet even asserts partitionValues.numFields == partitionSchema.length).

The survey supports the per-item choice anyway: (1) Iceberg's partition-spec evolution means files in a single scan can have different partition specs — constantsMap is computed per task from task.spec(), not once per scan — and Hive tables can likewise have per-partition descriptors; a single scan-level schema cannot express that. (2) FileOrFiles is already a heterogeneous list in Substrait (each item carries its own file_format), so per-item partition fields are consistent with the existing design.

Ordinal Expression.ReferenceSegment vs. name. Split 8 ordinal / 11 name / 1 field-id, and cleanly predicted by the carrier: positional where the carrier is a row or list, name where it is a map. Substrait's carrier already addresses fields positionally in this very message — the read projection is an Expression.MaskExpression — and names live in NamedStruct, so ordinal is the internally consistent choice. Name-keyed representations have also needed case-normalization hacks (Gluten's ClickHouse backend lower-cases both sides; Trino and Delta carry case rules), which positional addressing avoids.

Nested field support. Only Iceberg supports identity partitioning on a nested field today — PartitionSpec.Builder.identity(sourceName) resolves through schema.findField(sourceName), which accepts dotted paths, and readers key constants by field id at any depth. Everybody else is top-level only. So the nested capability ReferenceSegment provides is justified but not demanded: it comes for free from reusing ReferenceSegment and costs nothing.

Which fields come from the file vs. the path. Because partition fields are marked individually, Substrait avoids two conventions other engines were forced into: Spark and DataFusion require partition columns to be appended at the end of the schema, and BigQuery forbids overlap outright ("The Hive partitioning keys and the columns in the underlying files cannot overlap"). ClickHouse changed this rule in 25.8, moving partition columns from virtual to physical and requiring them in the table schema. Per-field marking is a genuine simplification.

File-constant metadata columns are deferred, and the shape generalizes. Four systems deliberately reuse the same mechanism for file-constant metadata columns: Comet ("Constant metadata columns … are, like partition columns, known before opening the file and constant for every row read from it. Reuse the same partition-value wire format"), DuckDB (the same constant map), Iceberg (constantsMap also carries _file, _spec_id, _row_id) and Dremio (PartitionValueType { VISIBLE, IMPLICIT, INVISIBLE }). Gluten's fork instead added a parallel NamedStruct.column_types enum (NORMAL_COL / PARTITION_COL / METADATA_COL / ROWINDEX_COL). Deferring metadata columns to a separate discussion is therefore fine — but (field reference, literal) is exactly what those four systems generalized to, so the deferral does not paint us into a corner.

---

A note on the Gluten citations

Gluten's vendored fork is being actively rebased onto the current spec as this is written, so the Gluten links above are pinned to commit 2a8bc38 rather than main. In flight at time of writing, all under GLUTEN-12597: apache/gluten#12650 (type system), apache/gluten#12700 (CrossRel -> NestedLoopJoinRel), apache/gluten#12704 (FetchRel), apache/gluten#12724 (AggregateRel.Grouping), apache/gluten#12727 (WindowRel -> ConsistentPartitionWindowRel), apache/gluten#12728 (TopNRel), apache/gluten#12746 (WriteRel). None of them touches the partition-column surface quoted here, which is still partitionColumn { string key; string value; } at field 17 plus the parallel NamedStruct.column_types enum.

That is the point, though: those two patches are among the remaining reasons the fork cannot go patch-free against core, and a core partition_fields is what would let them be dropped rather than renumbered. Gluten is cited here as evidence of the shape a real consumer needs, not as the justification — the survey's argument rests on the 19 other systems and on the three existing Substrait implementations that lose partition values today.

Source citations for all other systems link to main as of 2026-08-13. Happy to add any engine that reviewers think is missing.