Docker swarm - Cannot connect to kafka from outside

[njpsanghani]

original title: Cannot connect to kafka from outside

I have created this docker-compose file

running it using
docker stack deploy -c stack.yml zookeeper

The problem is the kafka cluster is not accessible from outside, what should i do.

i.e I can see the cluster using yahoo/kafka-manager, the zookeeper-s are visible hence i can create , list topic from outside, I cannot directly connect to the zookeeper cluster where needed like the console producer/consumer from host machine

version: '3.2'

services:
  zoo1:
    image: zookeeper
    restart: always
    hostname: zoo1
    ports:
      - 2181:2181
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
    networks:
     - rtls-cluster-network

  zoo2:
    image: zookeeper
    restart: always
    hostname: zoo2
    ports:
      - 2182:2181
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zoo3:2888:3888
    networks:
     - rtls-cluster-network

  zoo3:
    image: zookeeper
    restart: always
    hostname: zoo3
    ports:
      - 2183:2181
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=0.0.0.0:2888:3888
    networks:
     - rtls-cluster-network

  kafka1:
    image: wurstmeister/kafka:latest
    restart: always
    depends_on:
      - zoo1
      - zoo2
      - zoo3
    ports:
      - target: 9094
        published: 9092
        protocol: tcp
        mode: host
    environment:
      HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2"
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2182,zoo3:2183
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 3000
      KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: 'true'
      KAFKA_CONTROLLER_SHUTDOWN_ENABLE: 'true'
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_LOG_RETENTION_BYTES: -1
      KAFKA_LOG_RETENTION_DAYS: 2
      KAFKA_LOG_DIRS: /kafka/kafka-logs
    volumes:
      - /opt/data/kafka-1:/kafka/kafka-logs
    networks:
     - rtls-cluster-network

  kafka2:
    image: wurstmeister/kafka:latest
    restart: always
    depends_on:
      - zoo1
      - zoo2
      - zoo3
    ports:
      - target: 9094
        published: 9093
        protocol: tcp
        mode: host
    environment:
      HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2"
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2182,zoo3:2183
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 3000
      KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: 'true'
      KAFKA_CONTROLLER_SHUTDOWN_ENABLE: 'true'
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_LOG_RETENTION_BYTES: -1
      KAFKA_LOG_RETENTION_DAYS: 2
      KAFKA_LOG_DIRS: /kafka/kafka-logs
    volumes:
      - /opt/data/kafka-2:/kafka/kafka-logs
    networks:
     - rtls-cluster-network
  
  kafka3:
    image: wurstmeister/kafka:latest
    depends_on:
      - zoo1
      - zoo2
      - zoo3
    ports:
      - target: 9094
        published: 9094
        protocol: tcp
        mode: host
    environment:
      HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2"
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2182,zoo3:2183
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 3000
      KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: 'true'
      KAFKA_CONTROLLER_SHUTDOWN_ENABLE: 'true'
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_LOG_RETENTION_BYTES: -1
      KAFKA_LOG_RETENTION_DAYS: 2
      KAFKA_LOG_DIRS: /kafka/kafka-logs
    volumes:
      - /opt/data/kafka-3:/kafka/kafka-logs
    networks:
     - rtls-cluster-network

networks:
  rtls-cluster-network:
    

[sscaling]

This looks more like a Docker swarm question, rather than a kafka-docker image issue.

It would help to include more information, such as the steps that are failing for you (e.g. commands that run) along with their log output and logs from the brokers etc. Also probably information on your swarm setup (nodes, OS etc)

Looking at the port + hostname config - i'm not sure it looks correct. Given you're using the direct publishing of ports on every swarm node, i'm not sure how that relates to the value of HOSTNAME_COMMAND. This is the address that will be advertised via the metadata for the location of a specific partition. So in your case, you have advertised that they will be all on <host>:9094, but there is only one service (kafka3) that has port 9094 published.

[njpsanghani]

I am able to create the swarm without any issues issues.

The problem is i cannot
bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094 --topic test
run such commands from host machine where i have a another Kafka client.

Error

WARN [Producer clientId=console-producer] Error connecting to node 138f99a09487:9094 (id: 2 rack: null) (org.apache.kafka.clients.NetworkClient)
java.io.IOException: Can't resolve address: 138f99a09487:9094
	at org.apache.kafka.common.network.Selector.doConnect(Selector.java:235)
	at org.apache.kafka.common.network.Selector.connect(Selector.java:214)
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864)
	at org.apache.kafka.clients.NetworkClient.access$700(NetworkClient.java:64)
	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1035)
	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:920)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:508)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.UnresolvedAddressException
	at sun.nio.ch.Net.checkAddress(Net.java:101)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
	at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233)
	... 9 more

However i can do the below
bin/kafka-topics.sh --create --zookeeper localhost:2181,localhost:2182,localhost:2183 --replication-factor 1 --partitions 1 --topic test

This means that i am able to connect to the Zookeeper cluster from outside, however i cannot connect to the kafka cluster from outside.

Also I can connect to the cluster of zookeeper in yahoo/kafka-manager

[sscaling]

I'm not questioning how you create swarm. I'm just stating that it could be relevant for anyone trying to help you. For example, the way that you've structured the port allocation would indicated your running on a single node cluster (which would be advised against in a production environment. Also there is no pinning of broker to node, and therefore no pinning of broker to disk etc).

Regarding connectivity; As you can see from the error message, the client has been told to connect to 138f99a09487:9094 (meta data response). This is the HOSTNAME_COMMAND value + the port defined in advertised.listeners (i.e. OUTSIDE://_{HOSTNAME_COMMAND}:9094)

As you have only configured swarm node port mapping from ports 9092,9093 and 9094 this means that only one of your brokers is correctly addressable from the outside.

In a vastly simplified example, you have defined a comparable setup to the following:

• kafka1 - container id aaaa, advertised port 9094, published port 9093
• kafka2 - container id bbbb, advertised port 9094, published port 9094

advertised.listeners is configured to return the container's internal hostname (inside docker bridge / swarm overlay network - aaaa and bbbb hostname respectively).

port-forwarding is configured to map 9093->aaaa:9094' and 9094->bbbb:9094`.

However, as far as a Kafka client is concerned, during bootstrap it will retrieve the metadata information which will tell it to connect to aaaa:9094 and bbbb:9094 because that is what is specified in the advertised.listeners config. This is problematic because:

1. aaaa / bbbb are not resolvable publicly
2. port 9094 is only published for one of those addresses.

You will need to address at least both of those problems to move forward.

It might also be worth looking through some of the other docker-swarm tagged issues (https://github.com/wurstmeister/kafka-docker/issues?utf8=%E2%9C%93&q=is%3Aissue+label%3Adocker-swarm+) as other people have posted their configs.

[njpsanghani]

I am still not sure what should I do, How do i satisfy my purpose

[sscaling]

However, as far as a Kafka client is concerned, during bootstrap it will retrieve the metadata information which will tell it to connect to aaaa:9094 and bbbb:9094 because that is what is specified in the advertised.listeners config. This is problematic because:

• aaaa / bbbb are not resolvable publicly
• port 9094 is only published for one of those addresses.

As per my above comment, you will need to make sure your advertised host is resolvable publicly (currently it isn't) and also publish the correct ports (to match the ones that are being advertised).

  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - target: <ZZZ>
        published: <XXX>
        ... snip... 
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://<YYY>:<XXX>
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:<ZZZ>
      ... snip ...

• <YYY> needs to be resolvable outside the cluster
• <XXX> needs to match published
• <ZZZ> needs to match target

NOTE: <XXX> and <ZZZ> can be the same.

I would recommend starting with a 1 node cluster and getting that working to make sure you are familiar with the operational aspects. Use the relevant documentation for the version you are running.

However, this question is a more general Kafka configuration issues as opposed to a bug with the Kafka-docker image. I would suggest posting any further questions to the Kafka Mailing List which has a much broader support base. Also, for issues with Swarm you can ask on the Swarm forum.

Good luck!

p.s. Please re-open if you think there is a bug with the kafka-docker image. Thanks.

[sscaling]

Answering another ticket reminded me of this open PR: #377. It won't fix all your issues, but if may help you for the HOSTNAME_COMMAND to provide a resolvable address.

[enricodvn]

For docker swarm, I used the following command and it worked like a charm HOSTNAME_COMMAND: "docker info | grep 'Node Address: ' | cut -d' ' -f 4"

[minhtranes]

Hi @enricodvn and @njpsanghani ,

I encountered exactly the same issue, even I do tried the command you suggested. My env as following

environment:
  # HOSTNAME_COMMAND: "hostname -I | awk '{print $$2}'"
  HOSTNAME_COMMAND: "docker info | grep 'Node Address: ' | cut -d' ' -f 5"
  KAFKA_LISTENERS: DIL://:9092,DEL://:7092
  KAFKA_ADVERTISED_LISTENERS: DIL://:9092,DEL://_{HOSTNAME_COMMAND}:9092
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: DIL:PLAINTEXT,DEL:PLAINTEXT
  KAFKA_INTER_BROKER_LISTENER_NAME: DIL
  KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
  KAFKA_BROKER_ID: 1

Any idea ?