Description
Introduction
This ticket is a weekly-ish summary of interesting things happening in DataFusion. Note this is not a complete list (it is what I remember / can find). Please feel free to leave comments on this ticket about things that I may have missed or you think should get wider attention by the community. Follow on to #13630
Loosely inspired by https://this-week-in-rust.org/
Reminder, find new content (and please post some!) to
Community Highlights
- @Rachelint became a committer!
Theme: DataFusion Fever is Spreading
I think DataFusion is reaching an inflection point: It is now good enough that more than early adopters can and are building real, production systems using DataFusion. This is a great milestone 🎉 and I think the project is adjusting to this new reality.
One major theme we have been discussing in the last week or two is making upgrades easier. The recent pushes in 43.0.0 and 44.0.0 to clean up / complete projects such as StringView, window function migration, improved APIs, etc have caused significant downstream complications upgrading. Going forward as a community, we are discussing ways to improve this process.
I hope to write more on this topic
You can read more about this here
- [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) #13648
- [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB) #13525
New Blog Website
- @timsaucer has reworked the blog https://datafusion.apache.org/blog/ so that it now autopublishes when changes are made to https://github.com/apache/datafusion-site 🎉
Releases
- DataFusion python 43.0.0 was released: https://crates.io/crates/datafusion-python/43.1.0 🎉
- sqlparser release is in voting phase: Release sqlparser-rs version
0.53.0/ sqlparser_derive0.3.0datafusion-sqlparser-rs#1517 - arrow-rs major release Release arrow-rs / parquet major version
54.0.0(December 2024) arrow-rs#6342 - We are discussing DataFusion 44.0.0: Release DataFusion
44.0.0#13334 - DataFusion python release: chore: Upgrade to DataFusion 44 datafusion-python#972
Performance
The community loves a good benchmark challenge. We are off to a great start making h20 benchmark even faster, see
- [EPIC] Improved performance in H2O.ai benchmarks #13548
- @2010YOUY01 / @Dandandan and @jayzhan211 made
corrmore than 3x faster 🚀 - @Rachellint is also working on
mediandoing Improve performance ofmedianfunction #13550
I also also made a change with example to allow array reuse in functions, which adds to the 🚀 🔨 🧰
Thanks @dhegberg for a CSV loading benchmark
Also thanks to @richox, @Zhangli20, @tlm365 @jayzhan211 @Weijun-H, @comphead and @Dandandan for improving the speed of other functions
- Optimize performance of
initcapfunction (~2x faster) #13691 - Optimize performance of
character_lengthfunction #13696 - Improve substr() performance by avoiding using owned string #13688
- refactor: replace
VecwithIndexMapfor expression mappings inProjectionMappingandEquivalenceGroup#13675
🐛 fixes, and improvements
@onursatici has been on a tear along with @korowa @haohuaijin
-
fix: repartitioned reads of CSV with custom line terminator #13677
-
Thanks to @Eason0729 for Handle alias when parsing sql(parse_sql_expr) #12939
@findepi @jonahgao @comphead and others have been cleaning up the code 🧹
- Reveal implementing type and return type in simple UDF implementations #13730
- Report current operation when coercion fails #13628
- refactor: replace
OnceLockwithLazyLock#13641 - chore: macros crate cleanup #13685
- refactor: simplify the
make_udf_functionmacro #13712 - chore: clean up dependencies #13728
Unparser
We have been cranking away filling out plan --> SQL feature, thanks to @goldmedal
- Support unparsing
UNNESTplan toUNNESTtable factor SQL #13660 - Improve unparsing after optimize_projections optimization #13599
Hashbrown
@crepererum has been working to migrate our use of hashbrown to higher level APIs
- Upgrade to hashbrown 0.15.1: migrate from
hashbrown::raw::RawTabletohashbrown::hash_table::HashTable#13433 - refactor: change some
hashbrownRawTableuses toHashTable(round 3) #13658
Looking to get more involved? Try code review!
(can you see what I did there 🎣 )
DataFusion has a long history of community members contributing in all aspects of the project. Reviewing PRs is an especially great way to get introduced to the project, help the community and grow your own knowledge -- researching and understanding the code enough to review PRs also often inspires additional ideas for improvements.
We have docs about reviews. TLDR is: look for test coverage, if the change is understandable and well documented, and if the code can be improved. When you think the PR looks good to merge, try @ mentioning one of the committers.
Help wanted
- I would love to see the community offer additional help testing, triaging bugs helping to make DataFusion a more stable foundation for building systems
Please feel leave your own comments on this ticket if you are looking for help
Community
- Weekly Call
- Slack/Discord: info links