[client] Allow configuring scanner fetch parameters by Prajwal-banakar · Pull Request #417 · apache/fluss-rust

Prajwal-banakar · 2026-03-03T08:26:47Z

Purpose

Linked issue: close #391

Make log scanner fetch parameters configurable (max/min bytes and max wait time) instead of using hardcoded constants, so users can tune scan behavior via Config.

Brief change log

Removed hardcoded scanner fetch constants from client/table/scanner.rs.
Added fetch tuning fields to LogFetcher and wired them from crate::config::Config.
Updated fetch request construction to use configured values (including per-bucket max fetch cap).
Added/updated unit tests to validate configured fetch params are applied.

Tests

cargo fmt --all
cargo clippy -p fluss-rs --all-features
cargo test -p fluss-rs
Doc-tests: cargo test -p fluss-rs --doc

API and Format

API: Yes (adds/uses scanner fetch tuning knobs in Config; no breaking changes intended if defaults remain the same)
Storage format: No

Documentation

No new user-facing docs added (existing configuration/usage remains the same; defaults unchanged).

fresh-borzoni

@Prajwal-banakar Ty for the PR, left some comments, PTAL

fresh-borzoni · 2026-03-04T00:02:08Z


+    /// Maximum bytes per fetch response for LogScanner.
+    /// Default: 16777216 (16MB)
+    #[arg(long, default_value_t = DEFAULT_SCANNER_LOG_FETCH_MAX_BYTES)]


New config fields need Python bindings (FfiConfig + .pyi stub), C++ bindings, and website docs (configuration.md for both languages). Also missing from api-reference.md.

bindings and docs updated, PTAL!

Ty, ffi_converter.hpp wiring is still missing

Thanks for catching it, updated!

fresh-borzoni · 2026-03-04T00:02:46Z

-                            max_fetch_bytes: 1024 * 1024,
+                            max_fetch_bytes: self
+                                .fetch_max_bytes
+                                .min(DEFAULT_BUCKET_MAX_FETCH_BYTES),


This couples the total-response limit with the per-bucket cap, which is incorrect
Java has client.scanner.log.fetch.max-bytes-for-bucket for this

fresh-borzoni · 2026-03-04T00:05:09Z

        projected_fields: Option<Vec<usize>>,
    ) -> Result<Self> {
+        config
+            .validate_scanner_fetch()


mb we should validate during connection creation like we do with SASL fields?

fresh-borzoni · 2026-03-04T21:57:00Z

@Prajwal-banakar Ty, but I don't see cpp bindings have been wired

fresh-borzoni · 2026-03-04T21:57:53Z

    pub async fn new(arg: Config) -> Result<Self> {
        arg.validate_security()
            .map_err(|msg| Error::IllegalArgument { message: msg })?;
+        arg.validate_scanner_fetch() // ← add this


we don't need this comment

sorry, i just wired cpp bindings, PTAL!

fresh-borzoni

Ty, @Prajwal-banakar Left comments, PTAL

fresh-borzoni · 2026-03-05T17:43:17Z


+    /// Maximum bytes per fetch response for LogScanner.
+    /// Default: 16777216 (16MB)
+    #[arg(long, default_value_t = DEFAULT_SCANNER_LOG_FETCH_MAX_BYTES)]


Ty, ffi_converter.hpp wiring is still missing

fresh-borzoni · 2026-03-05T17:57:02Z

+                "scanner_log_fetch_min_bytes",
+                &self.scanner_log_fetch_min_bytes,
+            )
+            .field(


missing scanner_log_fetch_max_bytes_for_bucket

fresh-borzoni · 2026-03-05T17:57:28Z

    pub writer_batch_timeout_ms: i64,

+    /// Maximum bytes per fetch response **per bucket** for LogScanner.
+    /// Default: 1048576 (1MB) (or whatever DEFAULT_BUCKET_MAX_FETCH_BYTES is)


let's remove 'or whatever ...' part, it's inconsistent

fresh-borzoni

@Prajwal-banakar TY, left minor comment, PTAL

fresh-borzoni · 2026-03-06T09:07:15Z

+    ffi_config.scanner_log_fetch_wait_max_time_ms = config.scanner_log_fetch_wait_max_time_ms;
+    ffi_config.scanner_log_fetch_max_bytes_for_bucket = config.scanner_log_fetch_max_bytes_for_bucket;
+    ffi_config.writer_batch_timeout_ms = config.writer_batch_timeout_ms;
    ffi_config.writer_batch_timeout_ms = config.writer_batch_timeout_ms;


duplicated line

fresh-borzoni · 2026-03-06T09:15:33Z

                                ))
                            })?;
                    }
+                    "scanner.log.fetch-max-bytes" => {


Java uses scanner.log.fetch prefix, we should match it, also don't forget docs changes here

Prajwal-banakar · 2026-03-06T09:50:24Z

Hi @fresh-borzoni sorry for the repeated back-and-forth I'm still getting familiar with the codebase. Fixed the remaining issues, Thanks for the patience and thorough review!

fresh-borzoni

@Prajwal-banakar All good, I understand it takes time, so happy to review and help.

Thank you, LGTM

luoyuxia

@Prajwal-banakar Could you please rebase main branch, then i'll merge it.

Prajwal-banakar · 2026-03-11T17:37:07Z

Hi @luoyuxia i've rebased PTAL!

fresh-borzoni reviewed Mar 4, 2026

View reviewed changes

Prajwal-banakar requested a review from fresh-borzoni March 4, 2026 06:20

fresh-borzoni reviewed Mar 4, 2026

View reviewed changes

Prajwal-banakar requested a review from fresh-borzoni March 5, 2026 04:58

fresh-borzoni reviewed Mar 5, 2026

View reviewed changes

Prajwal-banakar requested a review from fresh-borzoni March 6, 2026 04:19

fresh-borzoni reviewed Mar 6, 2026

View reviewed changes

Prajwal-banakar requested a review from fresh-borzoni March 6, 2026 09:47

fresh-borzoni approved these changes Mar 6, 2026

View reviewed changes

luoyuxia approved these changes Mar 11, 2026

View reviewed changes

Prajwal-banakar added 5 commits March 11, 2026 15:52

Make scanner fetch params configurable

7dcc357

fix review comments

61b95ed

cpp binding wired

e5d330e

cpp binding wired

d73dd65

adressed review

15edf53

Prajwal-banakar force-pushed the configure-scanner branch from 02447a1 to 15edf53 Compare March 11, 2026 17:34

luoyuxia merged commit 8642ea4 into apache:main Mar 12, 2026
9 checks passed

Prajwal-banakar deleted the configure-scanner branch March 12, 2026 04:36

Conversation

Prajwal-banakar commented Mar 3, 2026

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fresh-borzoni commented Mar 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Prajwal-banakar commented Mar 6, 2026

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

Prajwal-banakar commented Mar 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants