Centralized Logging with EFK in Kubernetes

1. Introduction

Centralized logging is crucial to monitor and troubleshoot applications running in a Kubernetes cluster. The EFK stack (Elasticsearch, Fluentd, Kibana) provides a robust solution for this purpose, helping to aggregate logs from multiple sources and visualize them effectively.

2. EFK Stack Overview

The EFK stack consists of three main components:

• Elasticsearch: A distributed search and analytics engine that stores logs and provides powerful search capabilities.
• Fluentd: A data collector that gathers logs from various sources, processes them, and ships them to Elasticsearch.
• Kibana: A visualization tool that allows users to explore and analyze logs stored in Elasticsearch.

3. Installation Steps

Follow these steps to set up the EFK stack in your Kubernetes environment:

3.1 Install Elasticsearch

kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/1.8/examples/k8s/elasticsearch.yaml

3.2 Install Kibana

kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/1.8/examples/k8s/kibana.yaml

3.3 Install Fluentd

kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/1.8/examples/k8s/fluentd.yaml

4. Configuration

Once the EFK stack is installed, you need to configure Fluentd to collect logs from your applications. Below is an example configuration for Fluentd:

 


 # Fluentd configuration
 
   @type elasticsearch
   @log_level info
   host elasticsearch
   port 9200
   logstash_format true
 

 
   @type tail
   path /var/log/containers/*.log
   pos_file /var/log/fluentd-containers.log.pos
   tag kubernetes.*
   format json
 



 
 

5. Best Practices

• Use labels and annotations in your Kubernetes pods for better log categorization.
• Implement log rotation to prevent excessive disk space usage.
• Monitor the health of your Elasticsearch cluster to ensure performance and reliability.
• Regularly update your EFK stack components to benefit from the latest features and security patches.

6. FAQ