Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove inline optimization #20025

Draft
wants to merge 46 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
e34f0eb
concurrently fetch row from storage and refill cache
kwannoel Dec 2, 2024
414131c
add join key data types
kwannoel Dec 10, 2024
25a941a
wrap iterator in function
kwannoel Dec 10, 2024
14364d9
use in fetch matched rows
kwannoel Dec 10, 2024
6deafa8
same for delete
kwannoel Dec 10, 2024
0031217
more comments
kwannoel Dec 10, 2024
fe238e9
handle strong consistency requirement
kwannoel Dec 11, 2024
d921fea
unify branches
kwannoel Dec 12, 2024
b7e47a0
doc
kwannoel Dec 12, 2024
8afaab3
remove unused
kwannoel Dec 12, 2024
8742a75
replace
kwannoel Dec 12, 2024
f01e556
refactor inner row handling
kwannoel Dec 12, 2024
4dc407a
handle degrees implcitly
kwannoel Dec 12, 2024
7736df5
simplify
kwannoel Dec 12, 2024
7a890f1
unify caching
kwannoel Dec 12, 2024
c747be7
unify them in eq join
kwannoel Dec 12, 2024
1e152fd
remove old cache handling fn
kwannoel Dec 12, 2024
b6773fe
fix cache refill
kwannoel Dec 13, 2024
196e0ad
minor fix
kwannoel Dec 13, 2024
c68669c
optimize
kwannoel Dec 27, 2024
a776339
clean up
kwannoel Dec 27, 2024
45ee028
simplify with macro
kwannoel Dec 27, 2024
18de5fb
simplify
kwannoel Dec 27, 2024
5ea78a7
cleanup
kwannoel Dec 27, 2024
27d35fd
simplify
kwannoel Dec 27, 2024
6e2ae6a
increase hash join cache size
kwannoel Dec 27, 2024
0d41d32
make configurable
kwannoel Dec 27, 2024
9b3faef
handle null safe
kwannoel Dec 27, 2024
5b37e14
simplify
kwannoel Dec 28, 2024
4fcb636
use pretty assertions
kwannoel Dec 31, 2024
e40511a
fix bug
kwannoel Dec 31, 2024
cf98b91
fix another bug
kwannoel Dec 31, 2024
c4aa95c
fix bug
kwannoel Dec 31, 2024
2ce51ed
fmt
kwannoel Dec 31, 2024
232f46b
fix order key bug
kwannoel Dec 31, 2024
b972b41
fix config test
kwannoel Dec 31, 2024
f34b756
clean logs
kwannoel Dec 31, 2024
844a23e
handle degree row
kwannoel Jan 2, 2025
5d70a70
configure cache size in benchmark
kwannoel Jan 3, 2025
3582e72
inline handle match rows
kwannoel Jan 3, 2025
3644b4d
add comment for optimization
kwannoel Jan 3, 2025
2e0dacf
inline
kwannoel Jan 3, 2025
01407c8
fmt
kwannoel Jan 3, 2025
147d1f3
fmt
kwannoel Jan 3, 2025
0f5e5ab
drop inconsistency branch
kwannoel Jan 6, 2025
e18fca1
remove inline optimization
kwannoel Jan 4, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions src/common/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1150,6 +1150,11 @@ pub struct StreamingDeveloperConfig {
/// When true, all jdbc sinks with connector='jdbc' and jdbc.url="jdbc:postgresql://..."
/// will be switched from jdbc postgresql sinks to rust native (connector='postgres') sinks.
pub switch_jdbc_pg_to_native: bool,

/// Configure the system-wide cache row cardinality of hash join.
/// For example, if this is set to 1000, it means we can have at most 1000 rows in cache.
#[serde(default = "default::developer::streaming_hash_join_entry_state_max_rows")]
pub hash_join_entry_state_max_rows: usize,
}

/// The subsections `[batch.developer]`.
Expand Down Expand Up @@ -2113,6 +2118,11 @@ pub mod default {
pub fn switch_jdbc_pg_to_native() -> bool {
false
}

pub fn streaming_hash_join_entry_state_max_rows() -> usize {
// NOTE(kwannoel): This is just an arbitrary number.
30000
}
}

pub use crate::system_param::default as system;
Expand Down
1 change: 1 addition & 0 deletions src/config/example.toml
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ stream_exchange_connection_pool_size = 1
stream_enable_auto_schema_change = true
stream_enable_shared_source = true
stream_switch_jdbc_pg_to_native = false
stream_hash_join_entry_state_max_rows = 30000

[storage]
share_buffers_sync_parallelism = 1
Expand Down
1 change: 1 addition & 0 deletions src/stream/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ workspace-hack = { path = "../workspace-hack" }
assert_matches = "1"
criterion = { workspace = true, features = ["async_tokio", "async"] }
expect-test = "1"
pretty_assertions = "1"
risingwave_expr_impl = { workspace = true }
risingwave_hummock_sdk = { workspace = true }
risingwave_hummock_test = { path = "../storage/hummock_test", features = [
Expand Down
Loading
Loading