Kafka storage write and read path

[deniszh]

Hi All,
Could someone who know how Mimir with Kafka works help to realize what are write and read path for such use case?
I heard that if ingester-storgae enabled it helps with decoupling read path from write path.
So, if for usual Mimir we have:
Write

->Distributor->Ingester->S3

and Read

->Querier---->Ingester
        |
        +---->Store-gateway->S3

then what we have for Kafka use case?

I thought it was
Write

->Distributor------>Ingester->S3
             |
             +----->Kafka

and Read

->Querier<----Kafka
        |
        +---->Store-gateway->S3

Am I right?

[pracucci]

When using Kafka the write path is:

-> Distributor -> Kafka

The read path is

-> Query-frontend -> Querier -> Ingester <- Kafka
                             -> Store-gateway -> S3

In the Kafka architecture (internally called "ingest storage") you can think about the ingesters are read path components. Currently, ingesters are still responsible for building TSDB blocks uploaded to S3, but we're working on a new component called block-builder which is responsible to read from Kafka and build TSDB blocks. In the final architecture, ingesters will just be read caches for the most recent data, not available for being queried from TSDB blocks yet.