Apache Kafka Quickstart Tutorial

Apache Kafka is a fault tolerant publish-subscribe streaming platform that lets you process streams of records as they occur. This Kafka Quickstart Tutorial walks through the steps needed to get Apache Kafka up and running on a single Linux/Unix machine. In this tutorial we use Ubuntu and Kafka

Installing Java

Running Kafka requires Java. You can install Java from the command line using:

Downloading Apache Kafka

Download the desired version of Apache Kafka from a mirror site found at https://www.apache.org/dyn/closer.cgi?path=/kafka/. You can download Kafka through a browser or using wget:

Extracting Kafka Download Contents

In order to start using Kafka, you will need to extract the contents of the downloaded tarball. Using this command will extract the contents of the downloaded kafka_2.11- tarball into the current directory:

The extracted contents of this Kafka version will be a directory called kafka_2.11- We will consider this directory KAFKA_HOME. It is helpful to export the KAFKA_HOME variable as part of .bashrc or .bash_profile. This way it will be available to you whenever a new shell session is started.

From here you should be ready to start using Apache Kafka.

Starting Zookeeper

Kafka uses Zookeeper, so you must start a Zookeeper server before starting Kafka. Zookeeper comes as part of the Kafka download that was previously extracted, so there is no need to install it separately. You can start Zookeeper with the following commands:

Starting the Kafka Server

Now that a Zookeeper server is running, we can start our Kafka server. Zookeeper will be running in the foreground in the window it was started from. To start the Kafka server open a new terminal window and use the following commands:

Creating a Topic

Now that our Zookeeper and Kafka servers are running, we can now start using them. In a separate terminal window, use the command below to create the topic tutorial-test with only one partition and only one replica:

We can verify that the topic was created by listing all Kafka topics:

Producing Messages

Now that we have created a topic, we can now send messages to it. The kafka-console-producer will take messages from standard input and publish these messages to Kafka. Pressing <Enter> on your keyboard will send a message.

Here is an example of sending a message with the kafka-console-producer:

Consuming Messages

By starting a consumer we can consume messages sent to a Kafka topic. The kafka-console-consumer is included with the code we downloaded, and will read messages from a Kafka topic and write them to standard output. Adding the --from-beginning option will instruct the program to consume all available messages from the beginning of the topic.

This consumer will continue to consume messages as they are sent the the tutorial-test topic from our producer (in our example the kafka-console-producer):

5 thoughts on “Apache Kafka Quickstart Tutorial”

  1. Pingback: Creating a Simple Kafka Producer in Java - Big Datums

  2. Pingback: Creating a Simple Kafka Consumer - Big Datums

  3. Pingback: Sending Key Value Messages with the Kafka Console Producer - Big Datums

  4. Pingback: Writing Text File contents to Kafka with Kafka Connect - Big Datums

  5. Pingback: Writing Data from Apache Kafka to Text File - Big Datums

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">