-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add streaming support for reading rows from Bigtable #100
Conversation
Alright, removing the timeout parameter + 'a lifetime made things a lot nicer. What do you think? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just one suggestion and LGTM.
Hope the author likes it too. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks again, I think it looks great, just some minor comments above, if you have time to fix, otherwise I can merge it and release another version later.
Done! Thanks for the prompt review 👍🏻 |
🥳 If you're able to publish a release to crates.io, we'll be using this feature shortly :) |
@nrempel Thanks. I pushed in 0.2.15 just now. |
Add streaming support for reading rows from Bigtable
This PR adds streaming support for reading rows from Bigtable, allowing for more efficient processing of large result sets by streaming results as they arrive rather than waiting for the complete response.
Key Changes
Added new streaming methods to
BigTable
:stream_rows()
- Stream results from a standard row querystream_rows_with_prefix()
- Stream results from a prefix-based queryAdded new dependency:
futures-util = "0.3.31"
to both bigtable_rs and examples cratesImplemented streaming support in
read_rows.rs
withdecode_read_rows_response_stream()
Added a new example demonstrating streaming usage in
examples/src/stream.rs
Benefits
Note on Streaming and H2 Flow Control
The implementation leverages Tonic's H2 backpressure handling to create end-to-end flow control: consumers request data via
try_next()
, which pulls from Tonic only when ready, allowing the server to throttle based on client capacity. This native H2 flow control is particularly well-suited for Bigtable's variable-sized result sets and processing times, providing memory efficiency and network resilience without additional complexity.Example Usage
The new streaming API allows for processing results as they arrive: