Prometheus | Monitoring And Logging | Shell Scripting Tutorial

Introduction to Prometheus Integration

Prometheus is a popular open-source monitoring and alerting toolkit designed for reliability and scalability. Integrating Prometheus with shell scripts allows you to create custom metrics, automate monitoring tasks, and enhance your observability strategy.

Setting Up Prometheus

Before integrating Prometheus with shell scripts, you need to set up Prometheus on your server. Follow these steps:


# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.27.1/prometheus-2.27.1.linux-amd64.tar.gz

# Extract the tarball
tar xvfz prometheus-*.tar.gz

# Move the binaries to /usr/local/bin
sudo mv prometheus-2.27.1.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.27.1.linux-amd64/promtool /usr/local/bin/

# Create a Prometheus configuration file
sudo mkdir /etc/prometheus
sudo mv prometheus-2.27.1.linux-amd64/prometheus.yml /etc/prometheus/prometheus.yml

# Create a Prometheus data directory
sudo mkdir /var/lib/prometheus

# Create a Prometheus systemd service file
sudo bash -c 'cat << EOF > /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus \\
  --config.file=/etc/prometheus/prometheus.yml \\
  --storage.tsdb.path=/var/lib/prometheus/

[Install]
WantedBy=default.target
EOF'

# Reload systemd and start Prometheus
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Creating Custom Metrics with Shell Scripts

You can create custom metrics in Prometheus by using shell scripts. These metrics can then be exposed to Prometheus using a simple HTTP server. Here's an example shell script that generates a custom metric:


#!/bin/bash

# Custom metric for example counter
echo "# HELP example_counter A simple counter for demonstration purposes"
echo "# TYPE example_counter counter"
echo "example_counter $(($RANDOM % 100))"

Save this script as custom_metric.sh and make it executable:


chmod +x custom_metric.sh

Exposing Metrics to Prometheus

To expose the custom metrics to Prometheus, you can use a simple HTTP server. Here's an example using Python's built-in HTTP server:


#!/bin/bash

# Start a simple HTTP server to expose metrics
while true; do
  ./custom_metric.sh > /tmp/metrics
  sleep 10
done &
python3 -m http.server --directory /tmp 9100

Save this script as serve_metrics.sh and make it executable:


chmod +x serve_metrics.sh
./serve_metrics.sh

This script runs custom_metric.sh every 10 seconds and serves the metrics on port 9100.

Configuring Prometheus to Scrape Custom Metrics

Finally, configure Prometheus to scrape the custom metrics. Add the following job configuration to your prometheus.yml file:


scrape_configs:
  - job_name: 'custom_metrics'
    static_configs:
      - targets: ['localhost:9100']

Restart Prometheus to apply the configuration changes:


sudo systemctl restart prometheus

Conclusion

Integrating Prometheus with shell scripts allows you to create custom metrics and enhance your monitoring strategy. By using simple shell scripts and exposing metrics via an HTTP server, you can easily integrate with Prometheus and gain valuable insights into your system's performance.