Kafka Consumer POC

This is a small POC on Python Kafka Consumer using confluent_kafka library.

Pre-requisites

• Python 3
• Virtualenv
• Confluent Cloud Credentials

Install the dependencies

pip install -r requirements.txt

Environment Variables

export KAFKA_BOOTSTRAP_SERVERS=
export KAFKA_API_KEY=
export KAFKA_API_SECRET=

Consumer code

This code contains the KafkaConsumer class which reads a list of topics and consumes a configurable max_messages.

Create a instance of the KafkaConsumer class

Provide the topic list and the consumer configuration

consumer1 = KafkaConsumer(topic_list, config)

Consume the max messages from the topic list

Provide the max messages and timeout(in seconds) value to the consume method. It returns the consumed messages

messages = consumer1.consume(max_messages=5, timeout=10)