Skip to content

Troubleshooting: Kubernetes

Node IP change (the big one)

Full runbook: k8s/dke-mgmt/node-ip-change-runbook.md in the main repo — read it in full before improvising. Summary of the cascade, in order:

  1. etcd / kube-apiserver static pod manifests pin the old IP → bind failure, crashloop (bind: cannot assign requested address, or connection refused to 127.0.0.1:2379 once etcd itself is down). Fix: sed the correct IP into /etc/kubernetes/manifests/{etcd,kube-apiserver}.yaml. Never leave a .bak file inside the manifests directory — kubelet runs every file there as a static pod and will happily launch the stale one alongside the fixed one.
  2. kubelet/controller-manager/scheduler kubeconfigs must use the cluster's DNS name (server: https://<cluster>.devhome.cloud:6443), never a raw IP — the apiserver certificate's SANs contain the DNS name and the old IP only, so an IP-addressed kubeconfig fails TLS validation after any IP change even if the port is reachable. admin.conf is usually already correct; the other three are the ones that drift.
  3. systemctl restart kubelet, wait ~90s, verify etcd + apiserver reach Running (not just check the pod exists — confirm restart count isn't still climbing).
  4. Node object may still advertise the old InternalIP, and flannel's public-ip annotation follows it, breaking VXLAN between nodes. If a plain kubelet restart doesn't refresh it: 
    kubectl delete node <node>      # capture labels/taints first
systemctl restart kubelet       # re-registers with the current IP
kubectl label node <node> node-role.kubernetes.io/control-plane= \
    node.kubernetes.io/exclude-from-external-load-balancers=
kubectl taint node <node> node-role.kubernetes.io/control-plane=:NoSchedule
kubectl -n kube-flannel delete pods --all
    Not every occurrence needs this step — check whether the InternalIP already shows the current address before doing a full node deletion; a quieter kubelet restart sometimes refreshes it on its own.
  5. Vault's Kubernetes auth can 403 even with provably correct config — see Vault Kubernetes Auth.
  6. Uncordon if the shutdown playbook left nodes cordoned.

Why this doesn't get "permanently" fixed with a DHCP reservation

It was considered and declined — the intended long-term fix is decoupling the API endpoint from any node IP entirely via kube-vip's floating BGP-advertised VIP (see Terraform), which new clusters get by default. dke-mgmt inherits this when it's rebuilt, not via a live retrofit. In the meantime, a per-node DHCP reservation is a reasonable, narrow mitigation specifically for always-on management-plane nodes — it doesn't reverse the broader "no reservations" policy for general lab hosts, just closes the gap for the one cluster where an IP change is maximally disruptive.

Cluster-level follow-ups after any node-network change

  • Static NFS-backed PV (e.g. an image registry): the PV spec is immutable. If the NFS server's IP changed, delete the PV/PVC and hard-refresh the owning app — data is safe under Retain.
  • MetalLB: restart the controller pod after a pool change so services get reallocated, then update pfSense DNS host overrides to match — see DNS.
  • Dynamically-provisioned PVs embedding a previous NFS server IP go Released/dead — delete and let the provisioner recreate from the PVC.
  • bootstrap ApplicationSet on dke-mgmt is applied out-of-band — see Bootstrapping a New Cluster.