Alertmanager Tutorial

Introduction to Alertmanager

Alertmanager is a crucial component of the Prometheus ecosystem. It is responsible for managing alerts sent by Prometheus server and handling notifications to various channels. This includes grouping, silencing, inhibition, and sending notifications through different integrations such as email, Slack, and PagerDuty.

Installation

To get started with Alertmanager, you need to have it installed. You can download it from the official Prometheus website.

Extract the downloaded file:

Change into the directory:

Configuration

Alertmanager is configured through a YAML file, typically named alertmanager.yml. Below is a basic configuration example:

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'web.hook'

receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://example.com/hooks'
                

This configuration defines global settings and specifies a route for handling alerts. The receivers section defines where to send notifications.

Running Alertmanager

To start Alertmanager, run the following command from the directory containing alertmanager.yml:

Alertmanager will start and listen on port 9093 by default. You can access the web interface by navigating to http://localhost:9093 in your browser.

Integrating with Prometheus

Once Alertmanager is running, you need to configure Prometheus to send alerts to Alertmanager. You'll need to add an alerting section to your prometheus.yml configuration file:

alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093'] 
                

This tells Prometheus where to send alerts. Make sure to restart Prometheus after making changes to its configuration.

Alerting Rules

To actually send alerts, you need to define alerting rules in Prometheus. Here is an example of a simple alerting rule:

groups:
- name: example
  rules:
  - alert: HighCPUUsage
    expr: sum(rate(cpu_usage_seconds_total[5m])) by (instance) > 0.9
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High CPU usage detected"
      description: "CPU usage is over 90% for more than 5 minutes."
                

This rule triggers an alert when CPU usage exceeds 90% for 5 minutes.

Notifications

Alertmanager supports various notification channels. You can configure these in the receivers section of your alertmanager.yml file. For example, to send alerts to Slack:

receivers:
- name: 'slack-notifications'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/XXX/YYYY/ZZZ'
    channel: '#alerts'
                

Replace the api_url with your actual Slack webhook URL and specify the channel for notifications.

Conclusion

Alertmanager is a powerful tool for managing alerts in a Prometheus setup. By properly configuring it and integrating it with Prometheus, you can ensure that important alerts are sent to the right channels, allowing your team to react quickly to issues.

This tutorial covered the basics of installation, configuration, and integration with Prometheus. For more advanced features, such as grouping and silencing alerts, refer to the official Alertmanager documentation.