feat: support Nested Lists #627

gord02 · 2025-12-01T21:36:14Z

This PR is to add support for the Expression Nested List type.

Issue

Key Changes:

Substrait pojo to proto and back conversion
Substrait pojo to calcite and back conversion
Added new type in Isthmus to support Nested functions list during the conversion to Calcite from Substrait
Added code to not do a project remapping on an empty table

Testing:

added tests for roundtrip between substrait to proto
added tests for roundtrip between substrait and calcite

./gradlew test --tests io.substrait.isthmus.NestedExpressionsTest --debug-jvm

benbellick

Left a few comments. Thanks!

core/src/main/java/io/substrait/expression/Expression.java

core/src/main/java/io/substrait/expression/proto/ExpressionProtoConverter.java

core/src/main/java/io/substrait/expression/proto/ProtoExpressionConverter.java

core/src/test/java/io/substrait/type/proto/NestedListExpressionTest.java

benbellick

Just left a few more comments but the core/ part is looking great!

core/src/main/java/io/substrait/expression/Expression.java

core/src/main/java/io/substrait/expression/proto/ProtoExpressionConverter.java

core/src/test/java/io/substrait/type/proto/NestedListExpressionTest.java

benbellick

I think the core stuff looks great! I will try and take a pass at the isthmus stuff, but as I mentioned to you online, I am not as comfortable with that part of the codebase. If the changes there seem simple enough that I am comfortable approving, I will do so. Otherwise, I'll leave it to @vbarua or others to make the final judgement call 🙂

isthmus/src/main/java/io/substrait/isthmus/NestedFunctions.java

vbarua

Thanks for working this, its looking really good overall. I did have some comments, but they are minor suggestions. If you could take a look and respond to them, I think we can get this PR merge in this week easily 🙂

vbarua · 2025-12-10T05:17:32Z

core/src/main/java/io/substrait/expression/proto/ProtoExpressionConverter.java

+            nested.getList().getValuesList().stream().map(this::from).collect(Collectors.toList());
+        return ExpressionCreator.nestedList(nested.getNullable(), list);
+      default:
+        throw new IllegalStateException("Unimplemented nested type: " + nested.getNestedTypeCase());


minor: We can use UnsupportedOperationException here, which better matches your error message well 🙂

vbarua · 2025-12-10T05:22:27Z

core/src/main/java/io/substrait/expression/ExpressionCreator.java

  }

+  /**
+   * Creator a nested list expression with one or more elements.


Suggested change

* Creator a nested list expression with one or more elements.

* Creates a nested list expression with one or more elements.

vbarua · 2025-12-10T18:07:06Z

core/src/test/java/io/substrait/type/proto/NestedListExpressionTest.java

+  Expression.ScalarFunctionInvocation nonLiteralExpression = b.add(b.i32(7), b.i32(42));
+
+  @Test
+  void DifferentTypedLiteralsNestedListTest() {


Suggested change

void DifferentTypedLiteralsNestedListTest() {

void rejectNestedListWithElementsOfDifferentTypes() {

minor suggestion: if we include the expected behaviour in the test name, it's easier to see at at glance what its testing. Similar suggestions for test below.

vbarua · 2025-12-10T18:08:33Z

core/src/test/java/io/substrait/type/proto/NestedListExpressionTest.java

+  }
+
+  @Test
+  void EmptyNestedListTest() {


Suggested change

void EmptyNestedListTest() {

void rejectEmptyNestedListTest() {

vbarua · 2025-12-10T18:09:13Z

core/src/test/java/io/substrait/type/proto/NestedListExpressionTest.java

+  }
+
+  @Test
+  void SameTypedLiteralsNestedListTest() {


Suggested change

void SameTypedLiteralsNestedListTest() {

void acceptNestedListWithElementsOfSameType() {

Beyond the name suggestion for this test, it is a bit redundant because it doesn't really verify anything differently than your normal tests below do.

vbarua · 2025-12-10T18:26:49Z

isthmus/src/main/java/io/substrait/isthmus/SubstraitRelVisitor.java

            .map(this::toExpression)
            .collect(java.util.stream.Collectors.toList());

+    //    if there is no input fields, don’t put a remapping on it


minor/pedantic: small rewording + avoiding extra whitespace

Suggested change

// if there is no input fields, don’t put a remapping on it

// if there are no input fields, no remap is necessary

vbarua · 2025-12-10T18:29:26Z

isthmus/src/test/java/io/substrait/isthmus/NestedExpressionsTest.java

+  final Rel emptyTable = b.emptyScan();
+
+  @Test
+  void NestedListWithJustLiteralsTest() {


General Java style point. Tests are methods, so their names should be in camelCase. Only constructor functions that share the name of the class are capitalized

Suggested change

void NestedListWithJustLiteralsTest() {

void nestedListWithJustLiteralsTest() {

vbarua · 2025-12-10T18:29:33Z

isthmus/src/test/java/io/substrait/isthmus/NestedExpressionsTest.java

+  }
+
+  @Test
+  void NestedListWithNonLiteralsTest() {


Suggested change

void NestedListWithNonLiteralsTest() {

void nestedListWithNonLiteralsTest() {

vbarua · 2025-12-10T18:31:34Z

isthmus/src/test/java/io/substrait/isthmus/NestedExpressionsTest.java

+
+    RelNode relNode = substraitToCalcite.convert(project); //    substrait rel to calcite
+    Rel project2 = SubstraitRelVisitor.convert(relNode, extensions); // calcite to substrait
+    assertEquals(project, project2); // pojo -> calcite -> pojo


I would replace these 3 lines with just

assertFullRoundTrip(project);

which is a utility helper we have that checks more conversion than what you are checking now. Same for the test below.

vbarua · 2025-12-10T18:34:57Z

isthmus/src/test/java/io/substrait/isthmus/NestedExpressionsTest.java

+    RelNode relNode = substraitToCalcite.convert(project); //    substrait rel to calcite
+    Rel project2 = SubstraitRelVisitor.convert(relNode, extensions); // calcite to substrait
+    assertEquals(project, project2); // pojo -> calcite -> pojo
+  }


One more test that might be good to include in this file is a NestedList that includes as an expression a field from the input using a FieldReference.

asolimando

Left a few comments purely on the Isthmus side of things

asolimando · 2025-12-10T18:15:14Z

isthmus/src/main/java/io/substrait/isthmus/expression/NestedExpressionConverter.java

+  public Optional<Expression> convert(
+      RexCall call, Function<RexNode, Expression> topLevelConverter) {
+
+    if (!call.getOperator().getName().equals("NESTEDLIST")) {


Suggested change

if (!call.getOperator().getName().equals("NESTEDLIST")) {

if (!(call.getOperator() instanceof NestedListConstructor)) {

string-comparison is brittle and should be avoided

asolimando · 2025-12-10T18:16:57Z

isthmus/src/main/java/io/substrait/isthmus/SubstraitRelVisitor.java

            .map(this::toExpression)
            .collect(java.util.stream.Collectors.toList());

+    //    if there is no input fields, don’t put a remapping on it


Suggested change

// if there is no input fields, don’t put a remapping on it

// if there are no input fields, don’t put a remapping on it

nit

asolimando · 2025-12-10T18:19:19Z

isthmus/src/test/java/io/substrait/isthmus/NestedExpressionsTest.java

+  final Rel emptyTable = b.emptyScan();
+
+  @Test
+  void NestedListWithJustLiteralsTest() {


Suggested change

void NestedListWithJustLiteralsTest() {

void NestedListWithLiteralsTest() {

nit: but reads better IMO

asolimando · 2025-12-10T18:24:56Z

isthmus/src/main/java/io/substrait/isthmus/NestedListConstructor.java

+ * constructor creates a special type of SqlKind.ARRAY_VALUE_CONSTRUCTOR for lists that store
+ * non-literal expressions.


Conceptually and by looking at the tests (NestedListWithJustLiteralsTest), it seems this can indeed handle literal expressions, can you double check?

It can handle both literals and non-literals. I'll update the comment to avoid the confusion

asolimando · 2025-12-10T18:46:33Z

isthmus/src/main/java/io/substrait/isthmus/NestedListConstructor.java

+ * constructor creates a special type of SqlKind.ARRAY_VALUE_CONSTRUCTOR for lists that store
+ * non-literal expressions.
+ */
+public class NestedListConstructor extends SqlMultisetValueConstructor {


I guess you picked SqlMultisetValueConstructor instead of the more natural SqlArrayValueConstructor because CallConverters.java#L144 would then match on your NestedList and treat it as the regular SqlArrayValueConstructor (then invoke LiteralConstructorConverter.java#L32 which is not what we want here).

SqlMultisetValueConstructor is conceptually wrong as a multiset is radically different from an array/list in standard SQL, on top of that I imagine that tomorrow you might want to support something similar and hit the same issue you are trying to avoid here by using this class in the first place.

The impedance mismatch is that in SQL (and Calcite), arrays and lists are technically the same entity, while IIRC in Substrait they are treated as different entities (@benbellick can you confirm this?).

By looking at LiteralConstructorConverter, there is an implicit assumption that arrays store only literals, we go down that route without checking if elements in the array are really literals (LiteralConstructorConverter.java#L62).

It's probably enough to change LiteralConstructorConverter::toNonEmptyListLiteral to something like this (haven't tested it):

private Optional<Expression> toNonEmptyListLiteral( RexCall call, Function<RexNode, Expression> topLevelConverter) { List<Expression> expressions = call.operands.stream() .map(topLevelConverter) .collect(Collectors.toList()); // Check if all operands are actually literals if (expressions.stream().allMatch(e -> e instanceof Expression.Literal)) { return Optional.of(ExpressionCreator.list( call.getType().isNullable(), expressions.stream() .map(e -> (Expression.Literal) e) .collect(Collectors.toList()))); } return Optional.empty(); }

I suggest to extend SqlArrayValueConstructor (which, I know, extends SqlMultisetValueConstructor, but still), then fix LiteralConstructorConverter as suggested, so that we can continue with NestedExpressionConverter which comes just after, and we should be good

Just to make sure I am understanding it correctly, let me know your thoughts on the following scenarios: We want to ensure that the roundtrip of both a nested list with literals and non-literals are both returned to a nested list. If the literalConstructorConverter is run first on a list of just literals, then it would pass and then wouldn't be mapped back to a nested list. In the other case, where the nestedExpressionConverter is run first, the literal lists that were originally not a NestedList would become a nested list. Does the above account for this or is the difference important?

Also, is there a way to meaningfully extend the SqlArrayValueConstructor class? Its definition is bare with just a constructor to its parent type.

asolimando · 2025-12-10T19:01:19Z

isthmus/src/test/java/io/substrait/isthmus/NestedExpressionsTest.java

+
+class NestedExpressionsTest extends PlanTestBase {
+
+  protected static final SimpleExtension.ExtensionCollection defaultExtensionCollection =


I think we could add a few more tests.

mixed literal and non-literal expressions (IIRC this is the key scenario that justifies having both ListLiteral and NestedList), so here I'd test an "array" containing both literals and runtime expressions (like function calls) in the same list

field references: not sure it's legal in this context, but if it is, it would be nice to add a test for arrays containing references to input table columns (common use case for building dynamic arrays from row data)

ListLiteral preservation: due to what discussed here, I'd rather make sure creating a ListLiteral (all literals) round-trip it through Calcite, and assert that it remains a ListLiteral and was not converted to NestedList (especially if you accept my suggestion from the aforementioned comment)

Additional tests could be:

single-element lists,

nullable round-trip preservation through Calcite

different data types beyond integers and booleans

type coercion/type unification

gord02 added 3 commits November 19, 2025 13:54

feat: support Nested List Expressions

95c27c9

feat: adding substrait to calcite conversion

4b42b10

feat: making junit 5 tests package private

9e99091

gord02 force-pushed the gordon.hamilton/expressionNestedTypes branch from 18967e4 to 9e99091 Compare December 2, 2025 15:50

benbellick requested changes Dec 2, 2025

View reviewed changes

gord02 added 3 commits December 2, 2025 14:34

feat: fixing code based on PR review

5b277c9

feat: fixing empty list check

3ae2b62

feat: formatting

0e4c6fd

benbellick self-requested a review December 2, 2025 20:42

benbellick requested changes Dec 2, 2025

View reviewed changes

feat: fixing more tests and comments

5ad5fa1

benbellick reviewed Dec 2, 2025

View reviewed changes

isthmus/src/main/java/io/substrait/isthmus/NestedFunctions.java Outdated Show resolved Hide resolved

vbarua changed the title ~~Add support for Expression nested List Type~~ feat: support Nested Lists Dec 3, 2025

gord02 marked this pull request as draft December 9, 2025 14:31

gord02 added 2 commits December 9, 2025 13:11

fix: updating the return type in new sqlFunction

925c9ef

fix: resolving issues with nestedlist converter

b0a787d

gord02 marked this pull request as ready for review December 9, 2025 19:51

vbarua reviewed Dec 10, 2025

View reviewed changes

asolimando reviewed Dec 10, 2025

View reviewed changes

	* Creator a nested list expression with one or more elements.
	* Creates a nested list expression with one or more elements.

	void DifferentTypedLiteralsNestedListTest() {
	void rejectNestedListWithElementsOfDifferentTypes() {

	void EmptyNestedListTest() {
	void rejectEmptyNestedListTest() {

	void SameTypedLiteralsNestedListTest() {
	void acceptNestedListWithElementsOfSameType() {

	// if there is no input fields, don’t put a remapping on it
	// if there are no input fields, no remap is necessary

	void NestedListWithJustLiteralsTest() {
	void nestedListWithJustLiteralsTest() {

	void NestedListWithNonLiteralsTest() {
	void nestedListWithNonLiteralsTest() {

	if (!call.getOperator().getName().equals("NESTEDLIST")) {
	if (!(call.getOperator() instanceof NestedListConstructor)) {

	// if there is no input fields, don’t put a remapping on it
	// if there are no input fields, don’t put a remapping on it

	void NestedListWithJustLiteralsTest() {
	void NestedListWithLiteralsTest() {

		* constructor creates a special type of SqlKind.ARRAY_VALUE_CONSTRUCTOR for lists that store
		* non-literal expressions.


		class NestedExpressionsTest extends PlanTestBase {

		protected static final SimpleExtension.ExtensionCollection defaultExtensionCollection =

feat: support Nested Lists #627

Are you sure you want to change the base?

feat: support Nested Lists #627

Conversation

gord02 commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vbarua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asolimando left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gord02 Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gord02 commented Dec 1, 2025 •

edited

Loading

gord02 Dec 10, 2025 •

edited

Loading