Update results to Datafusion 46 #353
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AdamGS
commented
Apr 17, 2025
| CARGO_PROFILE_RELEASE_LTO=true RUSTFLAGS="-C codegen-units=1" cargo build --release | ||
| cd arrow-datafusion/ | ||
| git checkout 46.0.0 | ||
| CARGO_PROFILE_RELEASE_LTO=true RUSTFLAGS="-C codegen-units=1" cargo build --release --package datafusion-cli --bin datafusion-cli |
There was a problem hiding this comment.
Specifying package seems to help narrow the build scope by a bit
AdamGS
commented
Apr 17, 2025
| "tags": ["Rust", "column-oriented", "embedded", "stateless"], | ||
| "load_time": 0, | ||
| "data_size": 14779976446, |
There was a problem hiding this comment.
I think this was just copy-pasted from the single file, and the data isn't actually the same size.
AdamGS
commented
Apr 17, 2025
| @@ -31,7 +31,7 @@ cat queries.sql | while read -r query; do | |||
| # 2. each query contains a "Query took xxx seconds", we just grep these 2 lines | |||
| # 3. use sed to take the second line | |||
| # 4. use awk to take the number we want | |||
| RES=`datafusion-cli -f $CREATE_SQL_FILE /tmp/query.sql 2>&1 | grep "Elapsed" |sed -n 2p | awk '{ print $2 }'` | |||
| RES=$(datafusion-cli -f $CREATE_SQL_FILE /tmp/query.sql 2>&1 | grep "Elapsed" |sed -n 2p | awk '{ print $2 }') | |||
There was a problem hiding this comment.
this was weird, but the backticks just didn't work? this seems to work well
AdamGS
changed the title
|
@pmcgleenon do you mean just doing diff --git a/datafusion/queries.sql b/datafusion/queries.sql
-SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\\.)?([^/]+)/.*$', '\\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
+SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; |
Yes exactly that - the bug in the CLI that required adding the additional escape characters has been fixed. So they can be safely removed now |
|
thanks for the heads up! fixed it and reran everything. |
rschu1ze
approved these changes
Apr 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reran the benchmark for datafusion 46.0.0.
I ran into some minor issues with the scripts, so I fixed all the issues I ran into, including the README reflect the fact that since it was initially written tests moved to Ubuntu from Amazon Linux 2 (which seems to be the right move in the spirit of ClickBench)