Draining Nodes in Kubernetes

Introduction

In Kubernetes, draining a node is an essential maintenance activity that allows you to safely remove a node from service without affecting the workloads running in the cluster. It ensures that pods are gracefully terminated and rescheduled appropriately on other nodes.

Key Concepts

Key Definitions

Node: A worker machine in Kubernetes, which can be either a physical or virtual machine.
Pod: The smallest deployable unit in Kubernetes, which can contain one or more containers.
Draining: The process of evicting all pods from a node, ensuring they are rescheduled on other nodes.

Draining Nodes Process

The process of draining a node involves several steps to ensure that workloads are migrated safely. Below is a detailed step-by-step guide.

Identify the Node to Drain:
Run the Drain Command:

kubectl drain  --ignore-daemonsets

This command will cordon the node and evict all non-daemonset pods.

Monitor the Eviction Process:
Perform Maintenance:
Re-enable the Node:

kubectl uncordon

This command will allow scheduling to the node again.

Best Practices

Always use --ignore-daemonsets unless you want daemonsets to be affected.
Use --force with caution, as it can result in data loss if pods are not gracefully terminated.
Consider using --timeout to set a maximum wait time for pod eviction.
Regularly monitor node health and workloads to avoid unexpected downtime.

FAQ

What happens to the pods during a drain?

All pods running on the node will be evicted and Kubernetes will try to reschedule them on other available nodes.

Can I drain a node with running daemonsets?

Daemonsets are not affected by the drain command unless you use the --delete-local-data option, which can lead to data loss.

What is the difference between cordon and drain?

Cordon marks a node as unschedulable, preventing new pods from being scheduled on it, while drain evicts existing pods from the node.