Skip to content

Using StreamCuts and using readers from multiple process in the same reader group. #411

Open
@junglie85

Description

@junglie85

I'm trying to understand how the Rust client handles management of the last read position in a stream so that readers can continue from the last read position.

Suppose the following scenario:

  1. Client A writes data to s1.
  2. Client B creates reader group rg1 and reader r1.
  3. B reads from s1.
  4. B crashes.
  5. Client A continues to write data to s1.
  6. B restarts, creates reader group rg1 and reader r1.
  7. B starts reading from last read position in the stream.

If I understand correctly, I can start consuming a stream from a StreamCut, but what isn't clear to me from looking at the source code is how I would create a stream cut in the first place. Is there an API for this?

And if I do have a stream cut, is it my responsibility to persist it somehow or does Pravega take care of this for me?

It's also not clear to me how this relates to checkpointing?

If I wanted to add client C in another process/machine as another reader in reader group rg1, how can I do this? There isn't an API call for joining an existing reader group. Should I use the create_reader_group API call, possibly with an existing ReaderGroupConfig (which I assume it is my responsibility to serialise and transmit from B to C somehow) or is there a more correct approach to this?

Any help given in these areas is much appreciated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions