Welcome to my Kafka learning series! Here, I'll share my journey with Apache Kafka, from setting up the environment to mastering the CLI and beyond. For a deeper dive into all things Kafka, check out the extensive collection of articles on Baeldung's Kafka tag.

Kafka is a powerful stream processing platform developed by the Apache Software Foundation, and I'll guide you through the essentials of getting started and effectively using it.

Setting Up Kafka Using Docker on Mac

To quickly get Kafka up and running, Docker is an excellent choice. Here's a step-by-step guide:

I have created docker compose file for local setup.

Start a Single Zookeeper and Kafka Broker )

docker-compose -f single-zookeeper-single-kafka.yml up -d

• -d runs the Docker app in the background.
• -f specifies the file, allowing you to skip the default docker-compose.yml.

Check Service Status

docker-compose -f single-zookeeper-single-kafka.yml ps

Run Commands from Kafka Docker Container

docker exec -it kafka1 /bin/bash

Check Kafka Version

kafka-topics --version

Stop Kafka

docker-compose -f single-zookeeper-single-kafka.yml stop

Remove Kafka Completely

docker-compose -f single-zookeeper-single-kafka.yml down

Docker Exec with Sudo

docker exec -u root -t -i container_id /bin/bash

For a detailed setup guide, visit Conduktor's tutorial.

Using the Kafka CLI

Managing Topics

Create Topic

kafka-topics --bootstrap-server localhost:9092 --topic demo_topic --create --partitions 3 --replication-factor 1

List Topics

kafka-topics --bootstrap-server localhost:9092 --list

Use --exclude-internal to hide internal topics.

Describe Topic

kafka-topics --bootstrap-server localhost:9092 --describe --topic demo_topic

Alter Topic (Increase Partitions)

kafka-topics --bootstrap-server localhost:9092 --alter --topic demo_topic --partitions 4

Delete Topic

kafka-topics --bootstrap-server localhost:9092 --delete --topic demo_topic

For more on topic configurations, see the Kafka documentation.

Producer Operations

Push Messages

kafka-console-producer --bootstrap-server localhost:9092 --topic demo_topic

Push with Key

kafka-console-producer --bootstrap-server localhost:9092 --topic demo_topic --property parse.key=true --property key.separator=:

Explore more with Conduktor's producer CLI tutorial.

Consumer Operations

Consume Messages

kafka-console-consumer --bootstrap-server localhost:9092 --topic demo_topic

Consume from Beginning

kafka-console-consumer --bootstrap-server localhost:9092 --topic demo_topic --from-beginning --formatter kafka.tools.DefaultMessageFormatter --property print.timestamp=true --property print.value=true

Learn more about consumer CLI options on Conduktor.

Consumer Groups

Create Consumer Group

kafka-console-consumer --bootstrap-server localhost:9092 --topic topic_for_group_consumer --group my_first_application

Check Group Details

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my_first_application

Reset Offsets

kafka-consumer-groups --bootstrap-server localhost:9092 --group my_first_application --reset-offsets --to-earliest --execute --topic topic_for_group_consumer

Continue exploring consumer group management here.

Kafka Connectors

Kafka Connectors simplify the integration between Kafka and other systems. For a vast selection, visit the Connectors hub. Start learning about Kafka Connect here.

Kafka Application Development

Kafka isn't just about message passing; it's about building robust applications. Here are some great resources to get started:

• Kafka Programming Tutorials
• Java Kafka Programming
• Advanced Kafka Consumer with Java
• Spring Kafka on Baeldung

---

Stay tuned for more insights and tutorials on Kafka. Happy learning!

---