March 14, 2023

Kubernetes Control Plane and ETCD

Kubernetes control planes are a critical component of both on-premises and cloud-hosted clusters. They hold all of the cluster's metadata and provide a consistent view of cluster state, so understanding how they work together is essential for building highly available services.

A control plane hosts several services, including the API server, DNS, and ETCD. ETCD is particularly important because it's what allows control planes to maintain cluster consistency during failovers, through quorum-based operations. That's also why it's recommended to run an odd number of control planes — each with its own ETCD instance.

Still, this setup can create complications in certain failure scenarios — for example, when more than half of the control planes become unreachable, or when multiple control planes crash at once (say, 2 out of 3).

Listing ETCD Members

To see the current members of the ETCD cluster, run the following from a healthy control plane:

READY_ETCD_MEMBER="etcd-kubemaster1"
ETCDCTL="ETCDCTL_API=3 etcdctl --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --endpoints=https://127.0.0.1:2379"
kubectl -n kube-system exec -it "$READY_ETCD_MEMBER" -- sh -c "$ETCDCTL member list"

Replacing a Crashed Control Plane

Once you've identified the ID of the ETCD member that belongs to the crashed control plane, remove it so a replacement can be added cleanly:

READY_ETCD_MEMBER="etcd-kubemaster1"
ETCD_ID_TO_REMOVE="729aac77d384f8bd"
ETCDCTL="ETCDCTL_API=3 etcdctl --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --endpoints=https://127.0.0.1:2379"
kubectl -n kube-system exec -it "$READY_ETCD_MEMBER" -- sh -c "$ETCDCTL member remove $ETCD_ID_TO_REMOVE"