Supervisor (WCP) Operations
The vSphere Supervisor that backs TKC clusters has accumulated a lot of
hard-won operational knowledge, almost all of it hitting the same underlying friction
point: this lab's policy routing sends traffic that WCP assumes will arrive on the
management NIC (eth0) over the workload NIC (eth1) instead, because the Supervisor
VMs' own ip rule/routing table routes all 10.0.0.0/24 traffic — including
inter-VM management traffic — via eth1. Nearly every multi-host Supervisor bug below
traces back to this one root cause showing up in a different subsystem (etcd certs,
iptables, WCP's internal API routing).
etcd peer certs missing the workload IP as a SAN
Symptom: etcd can't elect a leader across hosts, kube-apiserver crashloops, Supervisor
stalls at "Configured Core Supervisor Services":
Because inter-host etcd peer connections arrive with a workload source IP (routed
via eth1), but WCP/kubeadm only ever put the management IP in the peer cert's SAN
list. Now automated — enable_supervisor.ps1's Invoke-WcpRbacRecovery detects each
CP VM's mgmt/workload IPs, regenerates peer.crt with the workload IP added (signed by
the existing etcd CA, idempotent), and restarts etcd on all 3 nodes simultaneously. Runs
at both the 8-minute CONFIGURING mark and the 5-minute ERROR mark.
iptables: etcd ports have no workload-NIC ACCEPT rule
Same root cause, different port. WCP's generated iptables template has eth0-only
ACCEPT for 2379/2380 (unlike 6443/10250, which already have eth1-scoped entries) — so
cross-host etcd traffic, arriving on eth1, falls through to the default DROP.
This is not a one-shot fix. Appending a plain iptables -A rule works once, then
gets silently reverted by a WCP startup-time process later. The durable fix (now in
ansible/supervisor-cluster-shutdown.yml Step 3c) has to (1) delete then insert at the
very top of the chain (-I INPUT 1), since a higher-priority eth0-only rule already
exists and order matters, and (2) run a background reassertion loop (36× every 5s) during
the startup window as a safety net. Also does not persist across reboots on Photon
OS — this step must re-run on every make start-tanzu.
A second bug compounds this: the supervisor VM root password rotates via guest
customization, and fetching it once upfront (rather than fresh per SSH attempt) causes a
uniform "Permission denied" across all 3 VMs if the fetch happens to race the rotation.
Fixed with an ssh_retry() wrapper that re-fetches the password fresh on each of up to 6
attempts, 5s apart.
Credentials for this SSH: use decryptK8Pwd.py (reads wcp.cluster_db_configs),
never decrypt_clustervm_pw.py (a different, non-working credential for this purpose).
Output is prefixed PWD:, not Password:. Needs
-o PreferredAuthentications=password -o KbdInteractiveAuthentication=no with sshpass
— without both, sshpass silently falls through to keyboard-interactive auth and fails
with exit 0 and no output.
WCP's internal cluster API VIP drifts — even with zero restarts
wcp.cluster_db_configs.IP (the VIP WCP itself uses for all internal API calls,
distinct from the two static HAProxy VIPs below) is dynamic and can be silently
reassigned by WCP mid-session, with the process having been running continuously for
55+ minutes with zero restarts. A fix that only re-syncs this IP at playbook-start time
cannot catch drift that happens hours later while everything is idle.
Durable fix: wcp-vip-watcher.service, a persistent systemd unit on the HAProxy VM
that polls the real IP (via decryptK8Pwd.py over SSH to vcsa) every 60 seconds,
compares it to whatever's currently bound in haproxy.cfg, and if different, updates
the frontend interface + anyip-routes.cfg + haproxy.cfg, validates with
haproxy -c, and reloads (not restarts, to avoid dropping connections). Installed/
refreshed on every make start-tanzu. If the HAProxy VM is ever rebuilt from the OVA
template (not just rebooted), this regenerates its own SSH trust automatically.
HAProxy must bind three distinct VIPs, not one
| VIP | Purpose | Static? |
|---|---|---|
10.0.0.145 |
WCP's internal management IP (master_mgmt_ip) — all WCP→apiserver calls go through this one |
Static, cached at first-enable time |
10.0.0.186 |
External kubectl vsphere login access (supervisor_control_plane_vip in Terraform) |
Static |
| dynamic | WCP's own internal cluster API VIP (see above) | Dynamic, watcher-managed |
Missing 10.0.0.145 specifically manifests as the Workload Management console stuck at
14/17 conditions, with WCP's log flooding
Request to apiserver failed ... Endpoint http://localhost:1080/external-cert/http1/10.0.0.145/6443/...
and ControlPlaneVMsConfigured oscillating true/false every ~8s. 10.0.0.170
(keepalived) is a third, unrelated VIP used only by the Supervisor's own kubeconfig —
don't confuse it with either of the two above.
Stale license asset blocks Kubernetes Identity permanently
A repeating Asset ... is already registered error in wcpsvc.log every 10s means a
stale license asset in vCenter's own VCDB (left over from a previous enable/disable
cycle) is blocking Kubernetes Identity (KI) creation — and without KI, the authorization
filter 500s every namespace operation (ki does not exist for supervisor domain-c27091).
A WCP service restart cannot fix this — it only clears in-memory state, not the VCDB
row. The supported fix is the full destroy/re-apply cycle
(terraform/vsphere/supervisor/destroy.sh then terraform apply — see
Terraform), which cascades-deletes the stale asset along with
everything else. Direct SQL deletion is possible but unsupported and risks orphaned FK
references; only use it if the destroy script itself fails.
kubectl get tkc can lie — trust kubectl get cluster instead
After a messy Supervisor restart, kubectl get tkc -A can show Ready=False on a
cluster that's actually healthy (kubectl get cluster -n tanzu shows Ready=True). The
deprecated TanzuKubernetesCluster controller
(vmware-system-tkg-controller-manager) caches stale ClusterBootstrap failure
conditions and doesn't re-read them once they clear. Fix:
kubectl annotate clusterbootstrap tkc-dev -n tanzu tkg.tanzu.vmware.com/add-missing-fields-from-tkr="true" --overwrite
then kubectl rollout restart deployment vmware-system-tkg-controller-manager -n svc-tkg-domain-c27091.
Always check get cluster, not get tkc, for authoritative status.
WCP service lifecycle: SSH, not the REST API
vmware.vmware_rest.appliance_services's start/stop calls for WCP silently return ok
without actually changing service state on this vCenter (confirmed via
service-control --status wcp staying unchanged either way). Use
ansible.builtin.raw: service-control --stop wcp / --start wcp over SSH to the vcsa
host instead — synchronous, blocks until genuinely complete. Also needs a login shell
specifically: service-control lives in /usr/local/sbin, which isn't on PATH for a
non-login shell, so a bare delegate_to: vcsa from another play won't find it — target
hosts: vcsa directly.
ArgoCD registration is Terraform-managed, same module as RKE2
The Supervisor's ArgoCD registration (https://10.0.0.170:6443, name supervisor) is
managed by terraform/vsphere/supervisor/argocd-register.tf, the same
argocd-register-clust module RKE2 uses (see RKE2 and
Bootstrapping a New Cluster). When migrating a
pre-existing manual registration into Terraform management, a straight terraform
import won't work if the live secret has a random-suffixed name (the old pre-1.24
argocd-manager-token-<suffix> pattern) — the module expects a fixed name. Delete the
manual registration and stale SA/CRB/secret first, then apply fresh.