Node Draining

Updated at 2018-12-31 00:46

Node draining means that you evict all pods from the node. The pods are relocated to other nodes. This is usually when you want to upgrade, modify or remove a node.

# find the node you want to empty of pods
kubectl get nodes

# start the process of evicting pods
# also cordons the node so it doesn't get more pods
kubectl drain <node name>

# __AFTER__ the above command has returned, you can remove 
# the node, upgrade it or do anything you want with it

# if you want to get the node to accept pods again, use uncordon
kubectl uncordon <node name>

Draining nodes is usually better to be done with clients, not kubectl. You submit eviction create requests through the REST API.

You should usually drain only one node at a time. If you want to drain multiple nodes in parallel, you must create a PodDisruptionBudget object.

PodDisruptionBudget (PDB) is application specific. PDB limits the number pods of a replicated application that are simultaneously down from voluntary disruptions. Cluster managers honor PDB automatically if they use evictions instead of manually deleting pods.

Deployment.spec.replicas: 5
# so the application should have 5 replicated pods online at all times,
# but PDB of 4 would allow voluntary disruptions to bring this down to 4

Distruption budget cannot prevent involuntary disruptions like hardware failure. But disruption budget does count those against the budget.

Distruption budget allows separating cluster owner and application owner roles. Useful if multiple teams are using the same cluster.