Elk Stack | Monitoring And Logging | Shell Scripting Tutorial

Introduction to ELK Stack

The ELK Stack is a powerful set of tools for searching, analyzing, and visualizing log data in real-time. ELK stands for Elasticsearch, Logstash, and Kibana. Integrating the ELK Stack with shell scripts allows you to automate the setup and configuration, making it easier to deploy and manage.

Installing Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine. Follow these steps to install Elasticsearch:


#!/bin/bash

# Download and install the public signing key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

# Install the apt-transport-https package
sudo apt-get install apt-transport-https

# Save the repository definition
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

# Update the package lists
sudo apt-get update

# Install Elasticsearch
sudo apt-get install elasticsearch

# Start and enable the Elasticsearch service
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

Installing Logstash

Logstash is a server-side data processing pipeline that ingests data from multiple sources simultaneously. Follow these steps to install Logstash:


#!/bin/bash

# Install Logstash
sudo apt-get install logstash

# Start and enable the Logstash service
sudo systemctl start logstash
sudo systemctl enable logstash

Installing Kibana

Kibana is a data visualization dashboard for Elasticsearch. Follow these steps to install Kibana:


#!/bin/bash

# Install Kibana
sudo apt-get install kibana

# Start and enable the Kibana service
sudo systemctl start kibana
sudo systemctl enable kibana

Configuring Logstash

Logstash needs to be configured to receive log data and send it to Elasticsearch. Create a configuration file named logstash.conf with the following content:


input {
    file {
        path => "/var/log/syslog"
        start_position => "beginning"
    }
}

filter {
    grok {
        match => { "message" => "%{SYSLOGLINE}" }
    }
    date {
        match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "syslog-%{+YYYY.MM.dd}"
    }
}

Save the file and start Logstash with the configuration:


sudo logstash -f /etc/logstash/conf.d/logstash.conf

Accessing Kibana

Once Kibana is installed and running, you can access it at http://localhost:5601. Follow these steps to set up your first dashboard:

Open Kibana in your web browser.
Go to the "Management" section and click on "Index Patterns".
Create a new index pattern that matches the indices created by Logstash (e.g., syslog-*).
Go to the "Discover" section to view the ingested logs.
Create visualizations and add them to a new dashboard.
Save the dashboard for future use.

Automating ELK Stack Setup with Shell Scripts

Here's a complete shell script to automate the setup of the ELK Stack:


#!/bin/bash

# Install Elasticsearch
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt-get update
sudo apt-get install elasticsearch
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

# Install Logstash
sudo apt-get install logstash
sudo systemctl start logstash
sudo systemctl enable logstash

# Install Kibana
sudo apt-get install kibana
sudo systemctl start kibana
sudo systemctl enable kibana

# Configure Logstash
cat <> /etc/logstash/conf.d/logstash.conf
input {
    file {
        path => "/var/log/syslog"
        start_position => "beginning"
    }
}

filter {
    grok {
        match => { "message" => "%{SYSLOGLINE}" }
    }
    date {
        match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "syslog-%{+YYYY.MM.dd}"
    }
}
EOT
sudo systemctl restart logstash

Save this script as setup_elk.sh and make it executable:


chmod +x setup_elk.sh
./setup_elk.sh

Conclusion

Setting up the ELK Stack with shell scripts automates the installation and configuration process, making it easier to deploy and manage. By integrating Elasticsearch, Logstash, and Kibana, you can efficiently collect, analyze, and visualize log data, enhancing your monitoring and logging capabilities.

Setting Up the ELK Stack with Shell Scripts