Getting Started With Apache Kafka

This tutorial will get us started with Apache Kafka on a Debian 8.

What Is Kafka?

Kafka is a new publish-subscribe messaging system that uses a distributed, partitioned, and replicated. It can scale horizontally without downtime, and achieves durability with messages being persisted on disk and replicated within the cluster to prevent data loss. Apparently, each broker can handle terabytes of messages without performance impact, but I have not tested this.

• For those of you that want to read more about the architecture, read the official getting started guide.
• For those of you interested in reading more about its performance, you may want to read "Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)". I

Requirements

• Install Java

Steps

Go to the downloads page and download the latest release which at the time of writing this tutorial is 0.9.0.1. Then extract the source code and navigate to within the extracted folder:

wget http://ftp.fau.de/apache/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgz
tar --extract --gzip --file kafka*
cd kafka_*

Kafka requires Zookeeper (another Apache product), in order to run. Setting up Zookeeper cluster is beyond the scope of this tutorial, so for now we will just have this instance run a zookeeper cluster of one. Luckily, the Kafka download provides an easy way to do this:

./bin/zookeeper-server-start.sh config/zookeeper.properties

Now we can start the kafka server.

./bin/kafka-server-start.sh config/server.properties

We now have a Kafka server running. Next time we'll learn about interfacing with it using PHP and setting up an actual cluster.