Getting Started with Kafka

Installing Kafka

Apache Kafka is a distributed streaming platform that can be used to build real-time streaming data pipelines and applications. This guide will walk you through the process of installing Kafka on your local machine.

Prerequisites

Before you begin, ensure you have the following installed on your system:

• Java 8+: Kafka requires Java to run. You can check if Java is installed by running java -version in your terminal. If Java is not installed, download and install it from the official Java website.
• Apache Zookeeper: Kafka uses Zookeeper to manage distributed brokers. The Kafka package includes Zookeeper, but it can also be installed separately if needed.

Step 1: Download Kafka

First, download the latest version of Kafka from the official website:

wget https://downloads.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz

Extract the downloaded tar file:

tar -xzf kafka_2.13-3.0.0.tgz
cd kafka_2.13-3.0.0

Step 2: Start Zookeeper

Kafka requires Zookeeper to manage its cluster. Start Zookeeper by running:

bin/zookeeper-server-start.sh config/zookeeper.properties

Zookeeper will start and listen on port 2181 by default.

Step 3: Start Kafka Server

In a new terminal, start the Kafka server by running:

bin/kafka-server-start.sh config/server.properties

The Kafka server will start and listen on port 9092 by default.

Step 4: Create a Topic

Kafka stores messages in topics. Create a new topic named test:

bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

You can verify that the topic was created successfully by listing all topics:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Step 5: Send Messages to the Topic

Open a new terminal and start a Kafka producer to send messages to the test topic:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>Hello, Kafka!
>This is a test message.

Step 6: Read Messages from the Topic

Open another terminal and start a Kafka consumer to read messages from the test topic:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

You should see the messages you produced earlier:

Hello, Kafka!
This is a test message.

Stopping Kafka and Zookeeper

To stop the Kafka server, press Ctrl+C in the terminal where Kafka is running. Similarly, to stop Zookeeper, press Ctrl+C in the terminal where Zookeeper is running.

If you prefer to stop them gracefully, use the following commands:

bin/kafka-server-stop.sh
bin/zookeeper-server-stop.sh

Conclusion

Congratulations! You have successfully installed and configured Kafka on your local machine. You have learned how to start Kafka and Zookeeper, create a topic, produce messages to the topic, and consume messages from the topic. Kafka is now ready to be used for building real-time streaming data pipelines and applications.