Can't make Kafka storage work

[deniszh]

Subj. Maybe someone can advise me what's wrong.
I tried docker compose from https://github.com/grafana/mimir/tree/main/development/mimir-ingest-storage, own docker compose setup or jsonnet - it's always the same - all components starts, reads return nothing, because distributor can't write anything, complaining about send data to partitions: DoBatch: InstancesCount <= 0.
Tried multiple distributors, multiple ingesters, with 2 zones or 3 zones - same error.
Kafka set up is similar to https://github.com/grafana/mimir/blob/main/development/mimir-ingest-storage/docker-compose.yml#L31-L102 - 3 node cluster with KRaft.
If I'm reading Jsonnet right for ingester storage we need to have ingester-storage enabled and kafka connection configured there - everything else should be the same?
Config from my docker compose is below:

# Do not use this configuration in production.
# It is for demonstration purposes only.
# Run Mimir in single process mode, with all components running in 1 process.
#target: all,alertmanager,overrides-exporter

# Configure Mimir to use Minio as object storage backend.
common:
  storage:
    backend: s3
    s3:
      endpoint: minio:9000
      access_key_id: mimir
      secret_access_key: supersecret
      insecure: true
      bucket_name: mimir

# ingest storage configuration.
ingest_storage:
  enabled: true
  kafka:
    address: kafka-1:9092
    topic: mimir-ingest
    last_produced_offset_poll_interval: 500ms
    startup_fetch_concurrency: 15
    ongoing_fetch_concurrency: 2
    auto_create_topic_enabled: true
    auto_create_topic_default_partitions: 100
    write_clients: 1
  migration:
    distributor_send_to_ingesters_enabled: false

ingester:
  track_ingester_owned_series: true
  partition_ring:
    min_partition_owners_count:       1
    min_partition_owners_duration:    10s
    delete_inactive_partition_after:  1m

# Blocks storage requires a prefix when using a common object storage bucket.
blocks_storage:
  storage_prefix: blocks
  tsdb:
    dir: /data/ingester

limits:
  ingestion_partitions_tenant_shard_size: 1
  native_histograms_ingestion_enabled: true
  ingestion_rate: 100000

# Use memberlist, a gossip-based protocol, to enable the 3 Mimir replicas to communicate
memberlist:
  join_members: 
    - mimir-b
    - mimir-r
    - mimir-s
    - mimir-i-a-1
    - mimir-i-a-2
    - mimir-i-b-1
    - mimir-i-b-2

ruler:
  rule_path: /data/ruler
  alertmanager_url: http://127.0.0.1:8080/alertmanager
  ring:
    # Quickly detect unhealthy rulers to speed up the tutorial.
    heartbeat_period: 2s
    heartbeat_timeout: 10s
  for_grace_period: 1s
  query_frontend:
    address: dns:///mimir-r-1:9095

alertmanager:
  data_dir: /data/alertmanager
  fallback_config_file: /etc/alertmanager-fallback-config.yaml
  external_url: http://localhost:9009/alertmanager

query_scheduler:
  service_discovery_mode: "ring"

overrides_exporter:
  ring:
    enabled: true
    wait_stability_min_duration: 30s

server:
  log_level: info

[pracucci]

Hi @deniszh, I can help you on this.

First of all, if you try the local development environment, by running ./development/mimir-ingest-storage/compose-up.sh (without any modification), does it work to you? It's not very clear to me whether it worked or not, from your message, but I think we should start from a working setup first, so you have a config reference.

If I'm reading Jsonnet right for ingester storage we need to have ingester-storage enabled and kafka connection configured there - everything else should be the same?

That's correct. To get it working you should just configure the ingest_storage block for all components (not all components technically need it, but it's just easier to configure it everywhere, doesn't hurt).

all components starts, reads return nothing, because distributor can't write anything, complaining about send data to partitions: DoBatch: InstancesCount <= 0

The error InstancesCount <= 0 means that the partitions ring is empty. The partitions ring is a new hash ring used by the ingest storage. Partitions are created by ingesters. Ingesters need to run with a specific naming, with the format <whatever>-<id> where <id> is a numeric ID representing the partition they own (e.g. 0, 1, 2, ...). From the snippet of config you shared, I can see mimir-i-a-1, so I guess these are the ingesters: if so, the naming looks correct.

The next step is to check the status of the partitions hash ring (and ingesters hash ring). What's the state of ingesters in web page /ingester/ring and what's the state of partitions in /ingester/partition-ring? Is there any error in the ingester logs about failing to register to the ring?

[deniszh]

Hi @pstibrany

Thanks a lot for your answer, it's directed me in right direction! I checked everything one more time, found some typos and looks like it's working in my compose! Thanks a lot!
Now I need to realize how it works and what's migration path from normal ingesters, but that's a different story.