Getting Started With Apache Kafka

This tutorial will get us started with Apache Kafka on a Debian 8.

What Is Kafka?

Kafka is a new publish-subscribe messaging system that uses a distributed, partitioned, and replicated. It can scale horizontally without downtime, and achieves durability with messages being persisted on disk and replicated within the cluster to prevent data loss. Apparently, each broker can handle terabytes of messages without performance impact, but I have not tested this.

Requirements

Steps

Go to the downloads page and download the latest release which at the time of writing this tutorial is 0.9.0.1. Then extract the source code and navigate to within the extracted folder:

wget http://ftp.fau.de/apache/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgz
tar --extract --gzip --file kafka*
cd kafka_*

Kafka requires Zookeeper (another Apache product), in order to run. Setting up Zookeeper cluster is beyond the scope of this tutorial, so for now we will just have this instance run a zookeeper cluster of one. Luckily, the Kafka download provides an easy way to do this:

./bin/zookeeper-server-start.sh config/zookeeper.properties

Now we can start the kafka server.

./bin/kafka-server-start.sh config/server.properties

We now have a Kafka server running. Next time we'll learn about interfacing with it using PHP and setting up an actual cluster.

Author

Programster

Stuart is a software developer with a passion for Linux and open source projects.

comments powered by Disqus
We are a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for us to earn fees by linking to Amazon.com and affiliated sites. More info.