Skip to content

Conversation

rishvin
Copy link
Contributor

@rishvin rishvin commented Sep 7, 2025

Which issue does this PR close?

Addresses Part of #1941

Rationale for this change

Introduces map_from_list which converts a ListArray<Struct<K, V> to MapArray.

This PR has been split from original PR #2221. Please see that for details.

What changes are included in this PR?

A new map_from_list physical function.

How are these changes tested?

Added unit tests.

@codecov-commenter
Copy link

codecov-commenter commented Sep 7, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.62%. Comparing base (f09f8af) to head (ded06b5).
⚠️ Report is 490 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2328      +/-   ##
============================================
+ Coverage     56.12%   57.62%   +1.49%     
- Complexity      976     1297     +321     
============================================
  Files           119      147      +28     
  Lines         11743    13497    +1754     
  Branches       2251     2390     +139     
============================================
+ Hits           6591     7777    +1186     
- Misses         4012     4451     +439     
- Partials       1140     1269     +129     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rishvin rishvin changed the title chore: [1941-Part3]: Introduce map_from_list scalar function feat: [1941-Part3]: Introduce map_from_list scalar function Sep 7, 2025
@comphead
Copy link
Contributor

comphead commented Sep 9, 2025

Sorry @rishvin it looks like some conflicts in there

@rishvin rishvin force-pushed the 1941-part3-introduce-map-from-list-function branch from 58c8744 to ded06b5 Compare September 10, 2025 03:34
@rishvin
Copy link
Contributor Author

rishvin commented Sep 10, 2025

Sorry @rishvin it looks like some conflicts in there

Thanks @comphead , resolved the conflicts. Here also I force pushed to change the commit message from chore to feat.

@comphead
Copy link
Contributor

I'm also reviewing this function in DF which might solve the same problem apache/datafusion#17456

@SparkApplicationMaster
Copy link

SparkApplicationMaster commented Sep 12, 2025

@comphead thanks for highlighting this PR
it has some interesting features like configuration values, curious if it can be achieved in runtime?
map_from_list solves map_from_entries spark function: https://spark.apache.org/docs/latest/api/sql/index.html#map_from_entries
actually the input StructArray is an array with two ListArray columns for keys and values
so map_from_entries can be solved reusing common code of map_from_arrays implementation
only StructArray unpacking is needed (it is a metadata-only operation)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants