Burrow | Tools And Frameworks

Introduction to Burrow

Burrow is a monitoring tool for Apache Kafka that provides consumer lag checking as a service. It helps you monitor the status of your Kafka consumers and alerts you if they fall behind or are not consuming messages from Kafka topics as expected.

Installing Burrow

To install Burrow, you need to have Go installed on your system. Follow these steps to install Burrow:

go get github.com/linkedin/Burrow

This command will fetch the Burrow source code and its dependencies. After fetching, you can build Burrow by navigating to the Burrow directory and running the build command:

cd $GOPATH/src/github.com/linkedin/Burrow

go install

Configuration

Burrow requires a configuration file to work correctly. The configuration file is written in TOML format. Here is a basic example of a Burrow configuration file:

[general]
logdir = "/var/tmp/burrowlogs"
logconfig = "/etc/burrow/logging.cfg"

[zookeeper]
hostname = "localhost"
port = 2181
timeout = 6

[kafka]
broker = "localhost:9092"
zookeeper = "localhost:2181"

Running Burrow

Once you have installed and configured Burrow, you can run it using the following command:

burrow --config-dir /path/to/config

This will start Burrow with the specified configuration directory. You should see logs indicating that Burrow has started and is monitoring your Kafka consumers.

Monitoring Kafka Consumers

Burrow provides a REST API to query the status of your Kafka consumers. You can use tools like curl or Postman to interact with the API. Here is an example of how to check the status of a specific consumer group:

curl -X GET http://localhost:8000/v3/kafka/local/consumer/{consumer_group}

{
  "error": false,
  "message": "consumer group status returned",
  "status": {
    "cluster": "local",
    "group": "my_consumer_group",
    "status": "OK",
    "partition_count": 3,
    "maxlag": {
      "partition": 1,
      "current_lag": 0,
      "end": {
        "offset": 100,
        "timestamp": 1609459200000,
        "lag": 0
      }
    },
    "total_lag": 0
  }
}

This response indicates that the consumer group my_consumer_group is operating normally and there is no lag.

Alerting

Burrow has built-in support for alerting when consumer groups fall behind. You can configure alerting by adding notifier sections to your Burrow configuration file. Here is an example of configuring an HTTP notifier:

[notify]
group = "default"
interval = 60

[notify.http]
url = "http://alertmanager.local/alert"
template = "default"

This configuration will send an alert to the specified URL if any consumer group falls behind.

Conclusion

Burrow is a powerful tool for monitoring Kafka consumers and ensuring they are functioning correctly. By following this tutorial, you should be able to install, configure, and run Burrow, as well as set up monitoring and alerting for your Kafka consumers.

Burrow Tutorial