Skip to content

Optimize planning / stop cloning Strings / Fields so much #18415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Optimize planning / stop cloning Strings / Fields so much #18415

wants to merge 3 commits into from

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Oct 31, 2025

Which issue does this PR close?

Rationale for this change

Avoid a bunch of clones / String copies during planning

What changes are included in this PR?

TBD

  1. TODO: update UpgradeGuide

Are these changes tested?

yes by CI

I will also run benchmarks

Are there any user-facing changes?

Yes, there are several API changes in DFSchema that now return FieldRef rather than Field which allows using Arc::clone rather than clone

@alamb alamb changed the title Alamb/stop cloning fields Optimize planning / stop cloning Strings / Fields so much Oct 31, 2025
@github-actions github-actions bot added sql SQL Planner logical-expr Logical plan and expressions core Core DataFusion crate substrait Changes to the substrait crate common Related to common crate labels Oct 31, 2025
alamb
alamb commented Oct 31, 2025
View reviewed changes
datafusion/expr/src/expr_schema.rs
metadata,
..
}) => {
let field = expr.to_field(schema).map(|(_, f)| f.as_ref().clone())?;
Copy link
Contributor Author

@alamb alamb Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these as_ref().clone() type methods are doing a deep clone on the Fields each time...

@alamb alamb mentioned this pull request Oct 31, 2025
Draft
@alamb
Copy link
Contributor Author

alamb commented Oct 31, 2025

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/stop_cloning_fields (074e485) to e65dafe diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_stop_cloning_fields
Results will be posted here when complete

alamb
alamb commented Oct 31, 2025
View reviewed changes
datafusion/common/src/dfschema.rs
/// Returns an immutable reference of a specific `Field` instance selected using an
/// offset within the internal `fields` vector
pub fn field(&self, i: usize) -> &Field {
pub fn field(&self, i: usize) -> &FieldRef {
Copy link
Contributor Author

@alamb alamb Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the changes in this file are the public API changes -- they return a reference to the Arc rather than what is in the Arc meaning the callsite can clone the arcs when needed

@alamb
Copy link
Contributor Author

alamb commented Oct 31, 2025

🤖: Benchmark completed

Details 

group                                                 alamb_stop_cloning_fields              main
-----                                                 -------------------------              ----
logical_aggregate_with_join                           1.00    623.2±2.35µs        ? ?/sec    1.01    631.0±5.28µs        ? ?/sec
logical_select_all_from_1000                          1.00     10.6±0.10ms        ? ?/sec    1.04     11.0±0.04ms        ? ?/sec
logical_select_one_from_700                           1.00    415.6±4.41µs        ? ?/sec    1.00    415.7±5.60µs        ? ?/sec
logical_trivial_join_high_numbered_columns            1.01    373.5±5.33µs        ? ?/sec    1.00    369.4±2.36µs        ? ?/sec
logical_trivial_join_low_numbered_columns             1.01    359.0±3.75µs        ? ?/sec    1.00    355.1±1.50µs        ? ?/sec
physical_intersection                                 1.00    833.7±7.62µs        ? ?/sec    1.00    834.4±3.02µs        ? ?/sec
physical_join_consider_sort                           1.00   1399.4±5.55µs        ? ?/sec    1.00   1394.1±9.37µs        ? ?/sec
physical_join_distinct                                1.00    348.1±1.43µs        ? ?/sec    1.00    348.6±4.03µs        ? ?/sec
physical_many_self_joins                              1.00      9.5±0.05ms        ? ?/sec    1.00      9.6±0.06ms        ? ?/sec
physical_plan_clickbench_all                          1.01    196.8±4.05ms        ? ?/sec    1.00    195.6±5.82ms        ? ?/sec
physical_plan_clickbench_q1                           1.00      2.4±0.03ms        ? ?/sec    1.06      2.5±0.08ms        ? ?/sec
physical_plan_clickbench_q10                          1.00      3.3±0.06ms        ? ?/sec    1.07      3.5±0.11ms        ? ?/sec
physical_plan_clickbench_q11                          1.00      3.5±0.05ms        ? ?/sec    1.10      3.9±0.14ms        ? ?/sec
physical_plan_clickbench_q12                          1.00      3.6±0.08ms        ? ?/sec    1.11      4.0±0.14ms        ? ?/sec
physical_plan_clickbench_q13                          1.00      3.2±0.04ms        ? ?/sec    1.09      3.6±0.11ms        ? ?/sec
physical_plan_clickbench_q14                          1.00      3.5±0.04ms        ? ?/sec    1.08      3.8±0.12ms        ? ?/sec
physical_plan_clickbench_q15                          1.00      3.4±0.05ms        ? ?/sec    1.06      3.6±0.10ms        ? ?/sec
physical_plan_clickbench_q16                          1.00      3.2±0.04ms        ? ?/sec    1.05      3.3±0.07ms        ? ?/sec
physical_plan_clickbench_q17                          1.00      3.3±0.06ms        ? ?/sec    1.05      3.4±0.09ms        ? ?/sec
physical_plan_clickbench_q18                          1.00      2.8±0.03ms        ? ?/sec    1.06      3.0±0.10ms        ? ?/sec
physical_plan_clickbench_q19                          1.00      3.7±0.06ms        ? ?/sec    1.09      4.0±0.11ms        ? ?/sec
physical_plan_clickbench_q2                           1.00      2.9±0.04ms        ? ?/sec    1.06      3.0±0.12ms        ? ?/sec
physical_plan_clickbench_q20                          1.00      2.5±0.03ms        ? ?/sec    1.09      2.8±0.09ms        ? ?/sec
physical_plan_clickbench_q21                          1.00      2.9±0.03ms        ? ?/sec    1.08      3.1±0.12ms        ? ?/sec
physical_plan_clickbench_q22                          1.00      3.8±0.11ms        ? ?/sec    1.01      3.8±0.14ms        ? ?/sec
physical_plan_clickbench_q23                          1.00      4.0±0.13ms        ? ?/sec    1.02      4.1±0.14ms        ? ?/sec
physical_plan_clickbench_q24                          1.00      4.5±0.12ms        ? ?/sec    1.02      4.6±0.15ms        ? ?/sec
physical_plan_clickbench_q25                          1.00      3.3±0.10ms        ? ?/sec    1.01      3.3±0.12ms        ? ?/sec
physical_plan_clickbench_q26                          1.02      3.1±0.10ms        ? ?/sec    1.00      3.0±0.11ms        ? ?/sec
physical_plan_clickbench_q27                          1.00      3.3±0.15ms        ? ?/sec    1.02      3.3±0.13ms        ? ?/sec
physical_plan_clickbench_q28                          1.00      4.1±0.15ms        ? ?/sec    1.01      4.2±0.19ms        ? ?/sec
physical_plan_clickbench_q29                          1.00      4.5±0.14ms        ? ?/sec    1.01      4.5±0.18ms        ? ?/sec
physical_plan_clickbench_q3                           1.00      2.8±0.03ms        ? ?/sec    1.07      3.0±0.10ms        ? ?/sec
physical_plan_clickbench_q30                          1.00     13.1±0.21ms        ? ?/sec    1.02     13.4±0.16ms        ? ?/sec
physical_plan_clickbench_q31                          1.00      4.0±0.12ms        ? ?/sec    1.04      4.2±0.15ms        ? ?/sec
physical_plan_clickbench_q32                          1.00      4.0±0.11ms        ? ?/sec    1.01      4.0±0.15ms        ? ?/sec
physical_plan_clickbench_q33                          1.00      3.5±0.12ms        ? ?/sec    1.01      3.5±0.16ms        ? ?/sec
physical_plan_clickbench_q34                          1.00      3.2±0.12ms        ? ?/sec    1.00      3.2±0.13ms        ? ?/sec
physical_plan_clickbench_q35                          1.00      3.3±0.09ms        ? ?/sec    1.00      3.3±0.12ms        ? ?/sec
physical_plan_clickbench_q36                          1.00      3.9±0.11ms        ? ?/sec    1.03      4.0±0.16ms        ? ?/sec
physical_plan_clickbench_q37                          1.00      4.2±0.13ms        ? ?/sec    1.02      4.3±0.18ms        ? ?/sec
physical_plan_clickbench_q38                          1.00      4.2±0.14ms        ? ?/sec    1.02      4.2±0.21ms        ? ?/sec
physical_plan_clickbench_q39                          1.00      4.1±0.13ms        ? ?/sec    1.02      4.1±0.16ms        ? ?/sec
physical_plan_clickbench_q4                           1.00      2.5±0.04ms        ? ?/sec    1.05      2.6±0.07ms        ? ?/sec
physical_plan_clickbench_q40                          1.00      4.8±0.14ms        ? ?/sec    1.02      4.9±0.19ms        ? ?/sec
physical_plan_clickbench_q41                          1.00      4.2±0.11ms        ? ?/sec    1.03      4.3±0.18ms        ? ?/sec
physical_plan_clickbench_q42                          1.00      4.1±0.12ms        ? ?/sec    1.03      4.2±0.15ms        ? ?/sec
physical_plan_clickbench_q43                          1.00      4.5±0.10ms        ? ?/sec    1.03      4.7±0.19ms        ? ?/sec
physical_plan_clickbench_q44                          1.00      2.7±0.06ms        ? ?/sec    1.02      2.8±0.11ms        ? ?/sec
physical_plan_clickbench_q45                          1.00      2.7±0.05ms        ? ?/sec    1.03      2.8±0.10ms        ? ?/sec
physical_plan_clickbench_q46                          1.00      3.2±0.08ms        ? ?/sec    1.02      3.2±0.09ms        ? ?/sec
physical_plan_clickbench_q47                          1.00      3.8±0.10ms        ? ?/sec    1.01      3.8±0.17ms        ? ?/sec
physical_plan_clickbench_q48                          1.00      4.6±0.12ms        ? ?/sec    1.04      4.8±0.20ms        ? ?/sec
physical_plan_clickbench_q49                          1.00      4.9±0.13ms        ? ?/sec    1.01      5.0±0.23ms        ? ?/sec
physical_plan_clickbench_q5                           1.00      2.7±0.04ms        ? ?/sec    1.06      2.9±0.09ms        ? ?/sec
physical_plan_clickbench_q50                          1.00      4.6±0.14ms        ? ?/sec    1.00      4.6±0.21ms        ? ?/sec
physical_plan_clickbench_q51                          1.00      3.3±0.09ms        ? ?/sec    1.00      3.3±0.15ms        ? ?/sec
physical_plan_clickbench_q6                           1.00      2.7±0.04ms        ? ?/sec    1.06      2.9±0.10ms        ? ?/sec
physical_plan_clickbench_q7                           1.00      2.4±0.03ms        ? ?/sec    1.07      2.6±0.10ms        ? ?/sec
physical_plan_clickbench_q8                           1.00      3.3±0.06ms        ? ?/sec    1.09      3.6±0.14ms        ? ?/sec
physical_plan_clickbench_q9                           1.00      3.2±0.05ms        ? ?/sec    1.08      3.4±0.12ms        ? ?/sec
physical_plan_tpcds_all                               1.00   1038.4±5.10ms        ? ?/sec    1.02  1055.2±15.34ms        ? ?/sec
physical_plan_tpch_all                                1.00     64.6±0.53ms        ? ?/sec    1.01     64.9±0.55ms        ? ?/sec
physical_plan_tpch_q1                                 1.00  1993.5±11.78µs        ? ?/sec    1.03      2.1±0.02ms        ? ?/sec
physical_plan_tpch_q10                                1.00      3.9±0.03ms        ? ?/sec    1.02      4.0±0.03ms        ? ?/sec
physical_plan_tpch_q11                                1.00      3.6±0.02ms        ? ?/sec    1.01      3.6±0.02ms        ? ?/sec
physical_plan_tpch_q12                                1.00  1815.6±11.28µs        ? ?/sec    1.00   1822.6±5.01µs        ? ?/sec
physical_plan_tpch_q13                                1.00  1451.8±20.14µs        ? ?/sec    1.02   1476.8±9.65µs        ? ?/sec
physical_plan_tpch_q14                                1.00  1946.3±10.46µs        ? ?/sec    1.02  1980.6±20.35µs        ? ?/sec
physical_plan_tpch_q16                                1.00      2.5±0.01ms        ? ?/sec    1.01      2.5±0.02ms        ? ?/sec
physical_plan_tpch_q17                                1.00      2.6±0.01ms        ? ?/sec    1.02      2.6±0.01ms        ? ?/sec
physical_plan_tpch_q18                                1.00      2.6±0.01ms        ? ?/sec    1.03      2.7±0.07ms        ? ?/sec
physical_plan_tpch_q19                                1.00      3.2±0.03ms        ? ?/sec    1.00      3.2±0.02ms        ? ?/sec
physical_plan_tpch_q2                                 1.00      6.1±0.09ms        ? ?/sec    1.00      6.1±0.06ms        ? ?/sec
physical_plan_tpch_q20                                1.00      3.2±0.02ms        ? ?/sec    1.01      3.2±0.02ms        ? ?/sec
physical_plan_tpch_q21                                1.00      4.2±0.03ms        ? ?/sec    1.01      4.3±0.03ms        ? ?/sec
physical_plan_tpch_q22                                1.00      2.8±0.02ms        ? ?/sec    1.01      2.8±0.01ms        ? ?/sec
physical_plan_tpch_q3                                 1.01      2.8±0.05ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_tpch_q4                                 1.00  1505.7±13.90µs        ? ?/sec    1.01   1517.6±9.68µs        ? ?/sec
physical_plan_tpch_q5                                 1.02      3.3±0.04ms        ? ?/sec    1.00      3.3±0.02ms        ? ?/sec
physical_plan_tpch_q6                                 1.00    859.8±4.47µs        ? ?/sec    1.01    866.9±4.01µs        ? ?/sec
physical_plan_tpch_q7                                 1.00      4.2±0.02ms        ? ?/sec    1.01      4.3±0.03ms        ? ?/sec
physical_plan_tpch_q8                                 1.00      5.5±0.05ms        ? ?/sec    1.01      5.6±0.10ms        ? ?/sec
physical_plan_tpch_q9                                 1.00      4.0±0.02ms        ? ?/sec    1.01      4.0±0.02ms        ? ?/sec
physical_select_aggregates_from_200                   1.00     16.9±0.10ms        ? ?/sec    1.01     17.0±0.10ms        ? ?/sec
physical_select_all_from_1000                         1.00     23.6±0.49ms        ? ?/sec    1.02     24.2±0.09ms        ? ?/sec
physical_select_one_from_700                          1.00   1074.4±4.18µs        ? ?/sec    1.00   1076.7±5.93µs        ? ?/sec
physical_sorted_union_order_by_10                     1.00     12.7±0.14ms        ? ?/sec    1.04     13.2±0.16ms        ? ?/sec
physical_sorted_union_order_by_50                     1.00    374.3±2.45ms        ? ?/sec    1.02    382.5±4.03ms        ? ?/sec
physical_theta_join_consider_sort                     1.00   1765.6±8.84µs        ? ?/sec    1.00  1760.4±14.43µs        ? ?/sec
physical_unnest_to_join                               1.00   1817.6±9.83µs        ? ?/sec    1.02  1858.4±16.44µs        ? ?/sec
physical_window_function_partition_by_12_on_values    1.00   1088.8±6.97µs        ? ?/sec    1.00   1092.7±4.53µs        ? ?/sec
physical_window_function_partition_by_30_on_values    1.00      2.2±0.01ms        ? ?/sec    1.00      2.2±0.01ms        ? ?/sec
physical_window_function_partition_by_4_on_values     1.00    664.3±5.65µs        ? ?/sec    1.01    669.0±5.94µs        ? ?/sec
physical_window_function_partition_by_7_on_values     1.00    810.6±3.49µs        ? ?/sec    1.01   815.0±12.95µs        ? ?/sec
physical_window_function_partition_by_8_on_values     1.00    867.6±6.07µs        ? ?/sec    1.01    878.6±7.25µs        ? ?/sec
with_param_values_many_columns                        1.00    129.0±0.85µs        ? ?/sec    1.00    128.8±0.73µs        ? ?/sec

@alamb
Copy link
Contributor Author

alamb commented Oct 31, 2025

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/stop_cloning_fields (074e485) to e65dafe diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_stop_cloning_fields
Results will be posted here when complete

@alamb
Copy link
Contributor Author

alamb commented Nov 1, 2025

🤖: Benchmark completed

Details 

group                                                 alamb_stop_cloning_fields              main
-----                                                 -------------------------              ----
logical_aggregate_with_join                           1.00    627.6±3.84µs        ? ?/sec    1.00    627.6±8.16µs        ? ?/sec
logical_select_all_from_1000                          1.00     10.8±0.06ms        ? ?/sec    1.04     11.1±0.06ms        ? ?/sec
logical_select_one_from_700                           1.01    418.2±3.54µs        ? ?/sec    1.00    415.9±2.80µs        ? ?/sec
logical_trivial_join_high_numbered_columns            1.00    372.0±2.24µs        ? ?/sec    1.00    371.7±5.18µs        ? ?/sec
logical_trivial_join_low_numbered_columns             1.00    357.0±3.65µs        ? ?/sec    1.00    355.9±1.08µs        ? ?/sec
physical_intersection                                 1.00    833.0±3.86µs        ? ?/sec    1.01    838.5±5.75µs        ? ?/sec
physical_join_consider_sort                           1.01  1403.1±15.26µs        ? ?/sec    1.00  1390.8±11.71µs        ? ?/sec
physical_join_distinct                                1.00    350.3±2.22µs        ? ?/sec    1.00    349.4±5.90µs        ? ?/sec
physical_many_self_joins                              1.01      9.8±0.06ms        ? ?/sec    1.00      9.7±0.07ms        ? ?/sec
physical_plan_clickbench_all                          1.07    195.4±2.81ms        ? ?/sec    1.00    183.4±1.76ms        ? ?/sec
physical_plan_clickbench_q1                           1.10      2.6±0.09ms        ? ?/sec    1.00      2.4±0.03ms        ? ?/sec
physical_plan_clickbench_q10                          1.00      3.3±0.04ms        ? ?/sec    1.08      3.6±0.13ms        ? ?/sec
physical_plan_clickbench_q11                          1.00      3.6±0.07ms        ? ?/sec    1.17      4.2±0.23ms        ? ?/sec
physical_plan_clickbench_q12                          1.00      3.7±0.06ms        ? ?/sec    1.10      4.1±0.17ms        ? ?/sec
physical_plan_clickbench_q13                          1.00      3.3±0.06ms        ? ?/sec    1.10      3.6±0.14ms        ? ?/sec
physical_plan_clickbench_q14                          1.00      3.5±0.06ms        ? ?/sec    1.12      4.0±0.14ms        ? ?/sec
physical_plan_clickbench_q15                          1.00      3.4±0.06ms        ? ?/sec    1.11      3.8±0.21ms        ? ?/sec
physical_plan_clickbench_q16                          1.00      3.2±0.05ms        ? ?/sec    1.05      3.4±0.11ms        ? ?/sec
physical_plan_clickbench_q17                          1.00      3.4±0.06ms        ? ?/sec    1.09      3.7±0.18ms        ? ?/sec
physical_plan_clickbench_q18                          1.00      2.9±0.04ms        ? ?/sec    1.03      3.0±0.09ms        ? ?/sec
physical_plan_clickbench_q19                          1.00      3.8±0.07ms        ? ?/sec    1.04      3.9±0.24ms        ? ?/sec
physical_plan_clickbench_q2                           1.11      3.1±0.11ms        ? ?/sec    1.00      2.8±0.02ms        ? ?/sec
physical_plan_clickbench_q20                          1.02      2.6±0.04ms        ? ?/sec    1.00      2.5±0.07ms        ? ?/sec
physical_plan_clickbench_q21                          1.09      3.1±0.31ms        ? ?/sec    1.00      2.9±0.04ms        ? ?/sec
physical_plan_clickbench_q22                          1.10      3.9±0.14ms        ? ?/sec    1.00      3.5±0.18ms        ? ?/sec
physical_plan_clickbench_q23                          1.02      4.2±0.13ms        ? ?/sec    1.00      4.1±0.30ms        ? ?/sec
physical_plan_clickbench_q24                          1.03      4.6±0.14ms        ? ?/sec    1.00      4.5±0.31ms        ? ?/sec
physical_plan_clickbench_q25                          1.13      3.4±0.14ms        ? ?/sec    1.00      3.0±0.03ms        ? ?/sec
physical_plan_clickbench_q26                          1.11      3.1±0.11ms        ? ?/sec    1.00      2.8±0.04ms        ? ?/sec
physical_plan_clickbench_q27                          1.03      3.4±0.12ms        ? ?/sec    1.00      3.3±0.14ms        ? ?/sec
physical_plan_clickbench_q28                          1.12      4.2±0.14ms        ? ?/sec    1.00      3.8±0.05ms        ? ?/sec
physical_plan_clickbench_q29                          1.11      4.6±0.15ms        ? ?/sec    1.00      4.1±0.07ms        ? ?/sec
physical_plan_clickbench_q3                           1.12      3.0±0.11ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q30                          1.01     13.2±0.14ms        ? ?/sec    1.00     13.0±0.35ms        ? ?/sec
physical_plan_clickbench_q31                          1.10      4.2±0.14ms        ? ?/sec    1.00      3.8±0.16ms        ? ?/sec
physical_plan_clickbench_q32                          1.10      4.2±0.15ms        ? ?/sec    1.00      3.8±0.13ms        ? ?/sec
physical_plan_clickbench_q33                          1.00      3.6±0.13ms        ? ?/sec    1.07      3.9±0.17ms        ? ?/sec
physical_plan_clickbench_q34                          1.00      3.2±0.13ms        ? ?/sec    1.08      3.4±0.12ms        ? ?/sec
physical_plan_clickbench_q35                          1.00      3.2±0.11ms        ? ?/sec    1.12      3.6±0.13ms        ? ?/sec
physical_plan_clickbench_q36                          1.00      3.8±0.06ms        ? ?/sec    1.16      4.4±0.15ms        ? ?/sec
physical_plan_clickbench_q37                          1.00      4.1±0.08ms        ? ?/sec    1.14      4.7±0.15ms        ? ?/sec
physical_plan_clickbench_q38                          1.00      4.1±0.08ms        ? ?/sec    1.17      4.7±0.11ms        ? ?/sec
physical_plan_clickbench_q39                          1.00      3.9±0.07ms        ? ?/sec    1.06      4.1±0.17ms        ? ?/sec
physical_plan_clickbench_q4                           1.04      2.6±0.05ms        ? ?/sec    1.00      2.5±0.02ms        ? ?/sec
physical_plan_clickbench_q40                          1.00      4.6±0.07ms        ? ?/sec    1.07      4.9±0.18ms        ? ?/sec
physical_plan_clickbench_q41                          1.00      4.1±0.07ms        ? ?/sec    1.05      4.3±0.18ms        ? ?/sec
physical_plan_clickbench_q42                          1.00      4.0±0.06ms        ? ?/sec    1.05      4.3±0.19ms        ? ?/sec
physical_plan_clickbench_q43                          1.00      4.5±0.08ms        ? ?/sec    1.05      4.7±0.17ms        ? ?/sec
physical_plan_clickbench_q44                          1.00      2.7±0.05ms        ? ?/sec    1.03      2.8±0.09ms        ? ?/sec
physical_plan_clickbench_q45                          1.01      2.7±0.04ms        ? ?/sec    1.00      2.7±0.10ms        ? ?/sec
physical_plan_clickbench_q46                          1.08      3.3±0.16ms        ? ?/sec    1.00      3.1±0.03ms        ? ?/sec
physical_plan_clickbench_q47                          1.12      4.0±0.13ms        ? ?/sec    1.00      3.6±0.04ms        ? ?/sec
physical_plan_clickbench_q48                          1.11      4.9±0.16ms        ? ?/sec    1.00      4.4±0.08ms        ? ?/sec
physical_plan_clickbench_q49                          1.12      5.2±0.15ms        ? ?/sec    1.00      4.7±0.07ms        ? ?/sec
physical_plan_clickbench_q5                           1.01      2.8±0.07ms        ? ?/sec    1.00      2.8±0.19ms        ? ?/sec
physical_plan_clickbench_q50                          1.12      4.8±0.14ms        ? ?/sec    1.00      4.2±0.06ms        ? ?/sec
physical_plan_clickbench_q51                          1.05      3.3±0.12ms        ? ?/sec    1.00      3.2±0.08ms        ? ?/sec
physical_plan_clickbench_q6                           1.00      2.8±0.11ms        ? ?/sec    1.11      3.2±0.40ms        ? ?/sec
physical_plan_clickbench_q7                           1.00      2.5±0.07ms        ? ?/sec    1.04      2.6±0.10ms        ? ?/sec
physical_plan_clickbench_q8                           1.00      3.5±0.09ms        ? ?/sec    1.12      3.9±0.20ms        ? ?/sec
physical_plan_clickbench_q9                           1.00      3.3±0.13ms        ? ?/sec    1.13      3.7±0.20ms        ? ?/sec
physical_plan_tpcds_all                               1.01   1067.4±9.05ms        ? ?/sec    1.00  1052.4±14.96ms        ? ?/sec
physical_plan_tpch_all                                1.00     64.6±0.47ms        ? ?/sec    1.03     66.7±0.81ms        ? ?/sec
physical_plan_tpch_q1                                 1.01      2.1±0.04ms        ? ?/sec    1.00      2.1±0.02ms        ? ?/sec
physical_plan_tpch_q10                                1.03      4.1±0.07ms        ? ?/sec    1.00      4.0±0.03ms        ? ?/sec
physical_plan_tpch_q11                                1.04      3.7±0.06ms        ? ?/sec    1.00      3.6±0.02ms        ? ?/sec
physical_plan_tpch_q12                                1.03  1862.9±44.70µs        ? ?/sec    1.00   1816.4±8.55µs        ? ?/sec
physical_plan_tpch_q13                                1.00  1466.6±11.01µs        ? ?/sec    1.00  1462.5±11.12µs        ? ?/sec
physical_plan_tpch_q14                                1.00  1954.2±15.86µs        ? ?/sec    1.02  1987.0±18.98µs        ? ?/sec
physical_plan_tpch_q16                                1.00      2.5±0.01ms        ? ?/sec    1.02      2.5±0.02ms        ? ?/sec
physical_plan_tpch_q17                                1.00      2.6±0.02ms        ? ?/sec    1.03      2.7±0.03ms        ? ?/sec
physical_plan_tpch_q18                                1.00      2.6±0.02ms        ? ?/sec    1.02      2.7±0.03ms        ? ?/sec
physical_plan_tpch_q19                                1.00      3.2±0.02ms        ? ?/sec    1.02      3.3±0.04ms        ? ?/sec
physical_plan_tpch_q2                                 1.05      6.4±0.09ms        ? ?/sec    1.00      6.0±0.10ms        ? ?/sec
physical_plan_tpch_q20                                1.00      3.2±0.02ms        ? ?/sec    1.03      3.3±0.05ms        ? ?/sec
physical_plan_tpch_q21                                1.00      4.2±0.03ms        ? ?/sec    1.04      4.4±0.06ms        ? ?/sec
physical_plan_tpch_q22                                1.00      2.8±0.03ms        ? ?/sec    1.04      2.9±0.04ms        ? ?/sec
physical_plan_tpch_q3                                 1.05      2.8±0.06ms        ? ?/sec    1.00      2.7±0.03ms        ? ?/sec
physical_plan_tpch_q4                                 1.01  1522.5±22.73µs        ? ?/sec    1.00   1508.0±5.57µs        ? ?/sec
physical_plan_tpch_q5                                 1.06      3.5±0.07ms        ? ?/sec    1.00      3.3±0.01ms        ? ?/sec
physical_plan_tpch_q6                                 1.00   866.6±10.60µs        ? ?/sec    1.00    863.8±9.50µs        ? ?/sec
physical_plan_tpch_q7                                 1.05      4.4±0.07ms        ? ?/sec    1.00      4.2±0.02ms        ? ?/sec
physical_plan_tpch_q8                                 1.06      5.8±0.08ms        ? ?/sec    1.00      5.5±0.05ms        ? ?/sec
physical_plan_tpch_q9                                 1.04      4.2±0.06ms        ? ?/sec    1.00      4.0±0.02ms        ? ?/sec
physical_select_aggregates_from_200                   1.01     17.3±0.09ms        ? ?/sec    1.00     17.1±0.55ms        ? ?/sec
physical_select_all_from_1000                         1.00     23.8±0.12ms        ? ?/sec    1.02     24.2±0.13ms        ? ?/sec
physical_select_one_from_700                          1.01  1088.8±10.38µs        ? ?/sec    1.00   1080.7±9.20µs        ? ?/sec
physical_sorted_union_order_by_10                     1.00     13.2±0.13ms        ? ?/sec    1.02     13.4±0.15ms        ? ?/sec
physical_sorted_union_order_by_50                     1.00    385.8±6.22ms        ? ?/sec    1.03    397.6±3.80ms        ? ?/sec
physical_theta_join_consider_sort                     1.00   1769.3±9.87µs        ? ?/sec    1.00  1765.2±13.13µs        ? ?/sec
physical_unnest_to_join                               1.00  1839.2±16.14µs        ? ?/sec    1.02   1873.8±9.34µs        ? ?/sec
physical_window_function_partition_by_12_on_values    1.00  1089.6±10.18µs        ? ?/sec    1.00   1093.8±7.14µs        ? ?/sec
physical_window_function_partition_by_30_on_values    1.01      2.3±0.03ms        ? ?/sec    1.00      2.3±0.01ms        ? ?/sec
physical_window_function_partition_by_4_on_values     1.00    666.7±7.30µs        ? ?/sec    1.00    665.5±3.62µs        ? ?/sec
physical_window_function_partition_by_7_on_values     1.00    817.0±7.47µs        ? ?/sec    1.00    818.3±4.13µs        ? ?/sec
physical_window_function_partition_by_8_on_values     1.00    879.2±6.38µs        ? ?/sec    1.00    877.6±5.22µs        ? ?/sec
with_param_values_many_columns                        1.00    127.7±0.76µs        ? ?/sec    1.00    127.9±0.76µs        ? ?/sec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

common Related to common crate core Core DataFusion crate logical-expr Logical plan and expressions sql SQL Planner substrait Changes to the substrait crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Optimize planning speed / stop cloning Strings / Fields so much

1 participant

@alamb