Querying 1TB on a laptop with Python dataframes – Ibis #9752
giscus[bot]
bot
-
Querying 1TB on a laptop with Python dataframes – Ibisthe portable Python dataframe library https://ibis-project.org/posts/1tbc/?utm_campaign=Data_Elixir&utm_source=Data_Elixir_494 |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
I would be curious to also see the case where we use Ibis with polars as backend, how much performance penalty would we pay in that case? |
Beta Was this translation helpful? Give feedback.
-
|
In theory, there should be zero overhead. In the past we've noticed that it was easy to defeat the Polars optimizer with redundant cast operations, but I've recently removed all unnecessary casts in the Polars backend. We haven't yet rerun this blog post code with this new version of the Ibis Polars backend. |
Beta Was this translation helpful? Give feedback.
-
|
you can find some earlier results of Polars directly vs Ibis with the Polars backend in the prior benchmarking post: https://ibis-project.org/posts/ibis-bench/ in general there was little overhead, though a few queries stood out. I'll re-run the benchmark with updated versions soon |
Beta Was this translation helpful? Give feedback.
you can find some earlier results of Polars directly vs Ibis with the Polars backend in the prior benchmarking post: https://ibis-project.org/posts/ibis-bench/
in general there was little overhead, though a few queries stood out. I'll re-run the benchmark with updated versions soon