Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add streaming support for reading rows from Bigtable #100

Merged
merged 2 commits into from
Nov 7, 2024

Conversation

nrempel
Copy link
Contributor

@nrempel nrempel commented Nov 4, 2024

Add streaming support for reading rows from Bigtable

This PR adds streaming support for reading rows from Bigtable, allowing for more efficient processing of large result sets by streaming results as they arrive rather than waiting for the complete response.

Key Changes

  1. Added new streaming methods to BigTable:

    • stream_rows() - Stream results from a standard row query
    • stream_rows_with_prefix() - Stream results from a prefix-based query
  2. Added new dependency:

    • futures-util = "0.3.31" to both bigtable_rs and examples crates
  3. Implemented streaming support in read_rows.rs with decode_read_rows_response_stream()

  4. Added a new example demonstrating streaming usage in examples/src/stream.rs

Benefits

  • More memory efficient for large result sets by processing rows as they arrive
  • Allows processing to begin before all results are received
  • Maintains consistent timeout handling with existing non-streaming methods
  • Preserves existing row parsing logic that is tested through conformance tests

Note on Streaming and H2 Flow Control

The implementation leverages Tonic's H2 backpressure handling to create end-to-end flow control: consumers request data via try_next(), which pulls from Tonic only when ready, allowing the server to throttle based on client capacity. This native H2 flow control is particularly well-suited for Bigtable's variable-sized result sets and processing times, providing memory efficiency and network resilience without additional complexity.

Example Usage

The new streaming API allows for processing results as they arrive:

❯ cargo run --package examples --bin stream
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.08s
     Running `target/debug/stream`
------------
key1
    [cf1:c1] "value1.v1" @ 1730754124325000
    [cf1:c1] "value1" @ 1730754120003000
------------
key2
    [cf1:c1] "value2" @ 1730754153781000
    [cf1:c1] "value2" @ 1730754120242000
------------
key3
    [cf1:c1] "value3" @ 1730754120486000

bigtable_rs/src/bigtable/read_rows.rs Outdated Show resolved Hide resolved
bigtable_rs/src/bigtable/read_rows.rs Outdated Show resolved Hide resolved
@nrempel
Copy link
Contributor Author

nrempel commented Nov 5, 2024

Alright, removing the timeout parameter + 'a lifetime made things a lot nicer.

What do you think?

Copy link

@paulodiovani paulodiovani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one suggestion and LGTM.
Hope the author likes it too. :)

bigtable_rs/src/bigtable.rs Outdated Show resolved Hide resolved
Copy link
Owner

@liufuyang liufuyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again, I think it looks great, just some minor comments above, if you have time to fix, otherwise I can merge it and release another version later.

@nrempel
Copy link
Contributor Author

nrempel commented Nov 6, 2024

Done! Thanks for the prompt review 👍🏻

@liufuyang liufuyang merged commit 719f3af into liufuyang:main Nov 7, 2024
2 checks passed
@nrempel
Copy link
Contributor Author

nrempel commented Nov 7, 2024

🥳

If you're able to publish a release to crates.io, we'll be using this feature shortly :)

@liufuyang
Copy link
Owner

@nrempel Thanks. I pushed in 0.2.15 just now.

@nrempel nrempel deleted the stream branch November 8, 2024 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants