Skip to content

Troubleshooting: Common Issues

Before improvising a fix, check whether this has already happened — most of the gnarliest issues in this lab have a committed runbook:

Symptom Likely cause Runbook
Cross-VLAN SMB/NFS/iSCSI transfer stalls after working briefly; pings fine Asymmetric routing on a dual-NIC TrueNAS box config/truenas/asymmetric-routing-fix.md
Docker/Apps container can't reach a service on its own host; SYN answered, no reply Same PBR gap, extended to docker bridges config/truenas/asymmetric-routing-fix.md
kubectl connection refused after a node reboot; etcd/apiserver crashlooping Node's DHCP IP changed; manifests still pin the old one k8s/dke-mgmt/node-ip-change-runbook.md
ClusterSecretStore InvalidProviderConfig/403 after cluster recovery Vault's kubernetes-auth plugin cached a stale client Vault Kubernetes Auth
Terraform destroy of a Tanzu/Supervisor stack hangs or leaves orphans vCenter WCP database state Terraform can't see Terraform
PXE boot stops working on one VLAN only Per-subnet Kea netboot settings, not a TFTP-server problem DHCP & PXE Boot
Internal DNS suddenly broken after a pfBlockerNG update FireHOL list action reset to Deny_Both pfBlockerNG
Multi-host Supervisor stuck CONFIGURING/ERROR; etcd can't elect a leader Workload-NIC routing hits etcd cert SAN/iptables gaps Supervisor (WCP) Operations
Helm chart pod crashes on NFS-backed PVC with an ownership/EINVAL error NFSv4 root-mapping + client-side idmapper caching TrueNAS Services
terraform apply reports a resource "broken" but validate/fmt pass clean Stray non-HCL notes in a hand-edited .tfvars Terraform

See Networking and Kubernetes for the deeper dives.

General diagnostic habits worth having

  • Read the memory/runbook before re-deriving anything. A past session likely already hit the same wall — check the relevant doc section here (and the repo's own committed runbooks) before spending an hour rediscovering it.
  • 2>/dev/null || true can hide a broken query, not just a harmless no-op. Several incidents in this lab trace back to a script error being silently swallowed for months — verify the underlying command actually does what its comment claims, especially against a schema/API that could have changed version.
  • Confirm the actual live state before trusting a service restart to have "worked." systemctl restart can report success while a stale in-memory cache (routing rules, auth plugin state) persists past the restart — verify the effect, not just the exit code.