Kafka Manager Tutorial
Introduction to Kafka Manager
Kafka Manager is an open-source tool developed by Yahoo to manage and monitor Apache Kafka. This tool provides an interface to manage multiple Kafka clusters, monitor topics, consumer groups, and much more. This tutorial will guide you through the installation, configuration, and usage of Kafka Manager.
Prerequisites
Before proceeding with the Kafka Manager installation, ensure you have the following prerequisites:
- Java 8 or higher installed on your system
- Apache Kafka installed and running
- Scala installed (Kafka Manager is a Scala-based application)
- sbt (Scala Build Tool) installed
Setting Up Kafka Manager
Follow these steps to set up Kafka Manager:
1. Clone the Kafka Manager repository from GitHub:
git clone https://github.com/yahoo/kafka-manager.git
2. Navigate to the Kafka Manager directory:
cd kafka-manager
3. Build the project using sbt:
./sbt clean dist
4. Navigate to the target directory:
cd target/universal
5. Extract the built package:
unzip kafka-manager-.zip
6. Start the Kafka Manager server:
cd kafka-manager-&& bin/kafka-manager
Replace <version> with the actual version number of the Kafka Manager you built.
Accessing Kafka Manager
Once the Kafka Manager server is running, you can access the web interface by navigating to http://localhost:9000 in your web browser.
Configuring Kafka Manager
After accessing the Kafka Manager web interface, follow these steps to configure your Kafka cluster:
1. Click on the "Cluster" menu and select "Add Cluster".
2. Fill in the required details:
- Cluster Name: A name for your Kafka cluster.
- Cluster Zookeeper Hosts: The Zookeeper hosts managing your Kafka cluster (e.g.,
localhost:2181). - Kafka Version: The version of Kafka you are using.
- Enable JMX Polling: (Optional) If you want to enable JMX polling for broker metrics.
3. Click "Save".
Your Kafka cluster should now be listed on the Kafka Manager dashboard.
Managing Kafka Topics
Kafka Manager allows you to create, delete, and view details of Kafka topics. Follow these steps to manage topics:
1. Click on the "Topics" menu and select "List".
2. To create a new topic, click on the "Create" button and fill in the topic details such as name, number of partitions, and replication factor.
3. To view details of an existing topic, click on the topic name from the list.
Here you can see details such as the number of partitions, replication factor, and current offsets.
4. To delete a topic, click on the "Delete" button next to the topic name.
Monitoring Consumer Groups
Kafka Manager provides a way to monitor consumer groups and their offsets. Follow these steps to monitor consumer groups:
1. Click on the "Consumers" menu and select "List".
2. You will see a list of consumer groups. Click on a consumer group to view details such as topic partitions and current offsets.
Conclusion
Kafka Manager is a powerful tool for managing and monitoring Apache Kafka clusters. It provides a user-friendly interface to perform various administrative tasks with ease. By following this tutorial, you should now have a working Kafka Manager setup and be able to manage your Kafka clusters efficiently.