Skip to content

Commit 5d5a276

Browse files
Fix map_query_sql benchmark duplicate key error (#18427)
Fix map_query_sql benchmark duplicate key error Description The build_keys() function was generating 1000 random keys from range 0..9999, which could result in duplicate keys due to the birthday paradox. The map() function requires unique keys, causing the benchmark to fail with: Execution("map key must be unique, duplicate key found: {key}") This fix ensures all generated keys are unique by: Using a HashSet to track seen keys Only adding keys to the result if they haven't been seen before Continuing to generate until exactly 1000 unique keys are produced Fixes #18421 Which issue does this PR close? Closes #18421 Rationale for this change The benchmark was non-deterministic: it could pass or fail depending on random key generation. With 1000 keys from a range of 9999 values, collisions are likely (~50% chance), making the benchmark unreliable. This change ensures uniqueness so the benchmark consistently succeeds and accurately measures map function performance. What changes are included in this PR? Added use std::collections::HashSet; import Modified build_keys() to: Track generated keys using a HashSet Only add keys if they are unique Continue generating until exactly 1000 unique keys are produced File changed: datafusion/core/benches/map_query_sql.rs Code changes: Added HashSet import at the top of the file Replaced simple loop with uniqueness-checking logic in build_keys() function Are these changes tested? The fix was verified by: Logic review: the HashSet approach guarantees uniqueness Code review: changes follow Rust best practices No linter errors The benchmark itself serves as the test — running cargo bench -p datafusion --bench map_query_sql should now complete without errors. Before this fix, the benchmark would fail with duplicate key errors in a significant portion of runs. Are there any user-facing changes? No user-facing changes. This is an internal benchmark fix that ensures the map_query_sql benchmark runs reliably. It does not affect the public API or any runtime behavior of DataFusion. --------- Co-authored-by: Jefffrey <[email protected]>
1 parent 7fa6378 commit 5d5a276

File tree

1 file changed

+6
-4
lines changed

1 file changed

+6
-4
lines changed

datafusion/core/benches/map_query_sql.rs

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18+
use std::collections::HashSet;
1819
use std::hint::black_box;
1920
use std::sync::Arc;
2021

@@ -33,11 +34,12 @@ use datafusion_functions_nested::map::map;
3334
mod data_utils;
3435

3536
fn build_keys(rng: &mut ThreadRng) -> Vec<String> {
36-
let mut keys = vec![];
37-
for _ in 0..1000 {
38-
keys.push(rng.random_range(0..9999).to_string());
37+
let mut keys = HashSet::with_capacity(1000);
38+
while keys.len() < 1000 {
39+
let key = rng.random_range(0..9999).to_string();
40+
keys.insert(key);
3941
}
40-
keys
42+
keys.into_iter().collect()
4143
}
4244

4345
fn build_values(rng: &mut ThreadRng) -> Vec<i32> {

0 commit comments

Comments
 (0)