Creating a Simple Kafka Consumer

Apache Kafka is a fault tolerant publish-subscribe streaming platform that lets you process streams of records as they occur. If you haven’t installed Kafka yet, see our Kafka Quickstart Tutorial to get up and running quickly.

In this post we will talk about creating a simple Kafka consumer in Java.

Kafka Consumer Code

The example below shows creating a Kafka consumer object and using it to consume messages from the my-topic topic.

In this example we are setting common properties like group.id which is the ID of the Kafka consumer group, and enable.auto.commit which is used for automatic commits.

This consumer creates an infinite loop, and each time through will use consumer.poll(100) to consume up to 100 records. The consumer prints each record’s offset and content.

Running the Kafka Consumer Example

In order to run this example, we need a Zookeeper server and a Kafka server running. For an example of how to do this see our Kafka Quickstart Tutorial to get up and running.

Once we have Zookeeper and Kafka running we can create the my-topic topic:

We are now ready to execute our code from the command line or from directly within an IDE and start consuming Kafka messages. Below is an example of running this code on the command line from our bigdatums-kafka-1.0-SNAPSHOT.jar JAR file.

Now that our consumer is running, we will test it by sending some sample messages to Kafka. We can do this in a separate terminal window with the kafka-console-producer. This is a program that is included with Kafka will take input from the command line (STDIN) and send it as messages to Kafka.

As these messages are sent to Kafka, our consumer will print the following lines:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">