Skip to content

Conversation

@jizezhang
Copy link
Contributor

@jizezhang jizezhang commented Nov 4, 2025

Which issue does this PR close?

Rationale for this change

This PR is intended to fix return type mismatch of spark array when inner data type is LargeList, e.g.

query error
SELECT array(arrow_cast(array(1), 'LargeList(Int64)'))
----
DataFusion error: Internal error: Function 'array' returned value of type 'LargeList(Field { name: "element", data_type: LargeList(Field { data_type: Int64, nullable: true }), nullable: true })' while the following type was promised at planning time and expected: 'List(Field { name: "element", data_type: LargeList(Field { data_type: Int64, nullable: true }), nullable: true })'.
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues

What changes are included in this PR?

  • Return List regardless of whether inner data type is LargeList or not. This aligns with the behavior of datafusion make_array function.
  • Remove return_field_from_args as return_type is already defined and is invoked internally.

Are these changes tested?

Yes

Are there any user-facing changes?

No.

@github-actions github-actions bot added sqllogictest SQL Logic Tests (.slt) spark labels Nov 4, 2025
@jizezhang jizezhang closed this Nov 4, 2025
@jizezhang jizezhang deleted the spark-array-fix branch November 4, 2025 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant