Network Design
Interfaces
pfSense's LAN interface (ix0, untagged/native) is plain LAN — described as
"Clients" in the pfSense config — not a tagged VLAN. Two tagged VLANs trunk over the
same physical ix0 port, plus a third for out-of-band hardware management:
| Interface | Purpose | Subnet | Tagged? |
|---|---|---|---|
LAN (ix0) |
Clients | 10.10.10.0/24 | No — native/untagged |
VLAN 20 (ix0.20, opt1) |
Management / Storage | 10.10.20.0/24 | Yes |
VLAN 30 (ix0.30, opt2) |
VM Workloads | 10.10.30.0/24 | Yes |
VLAN 200 (ix0.200) |
ILO Network (out-of-band hardware mgmt) | — | Yes |
Verify against the pfSense config before trusting this table
This is sourced directly from config/pfsense/pfsense-config.xml's <vlans> and
<interfaces> sections in the main repo — if pfSense gets reconfigured, re-check
that file rather than assuming this table stayed current.
The management/storage VLAN is deliberately untagged/native on each ESXi host's
UniFi switch port (the "Management" port profile) so that vmk0 lands on it directly
via DHCP with no separate storage vmkernel needed. A tagged VLAN 20 portgroup on these
ports would actually break traffic — the switch drops frames tagged with its own native
VLAN.
Physical NIC allocation pattern
Each ESXi host follows the same pattern where hardware allows: one NIC stays on the
default vSwitch0 for management (VLAN 20, DHCP), and a second physical NIC is
dedicated as the uplink for the VLAN 30 workload VDS (VM-Workloads-vDS). Which
specific vmnic plays which role is host-specific and must be verified against
vCenter's Physical Adapters page before any Terraform apply that touches host
networking — see Compute & Storage for the current mapping.
Two DHCP/routing gotchas worth internalizing
1. Asymmetric routing on dual-NIC hosts (pfSense PBR)
Both TrueNAS boxes (and any future dual-NIC host) sit on two directly-connected
subnets (LAN and VLAN 20). Without policy-based routing, a reply to cross-subnet
traffic exits whichever NIC the kernel's default route picks — not necessarily back
through pfSense — and pfSense's if-bound state policy silently drops it. This looked
like a NIC or pfSense bug for a long time before the actual cause was found; it broke
SMB writes, ESXi iSCSI/VMFS (journal timeouts), and — more recently — a Docker
container trying to reach a service on its own host.
The fix is a pbr-mgmt<vlan>.service systemd unit per box that adds ip rule from
<ip> lookup <table> policy routes with the matching per-source default gateway.
Full writeup, current unit files, and the Docker-bridge extension are in the repo:
config/truenas/asymmetric-routing-fix.md. Read that before touching routing on either
TrueNAS box.
2. No DHCP reservations for Kubernetes nodes (by design, with a real cost)
Cluster nodes get plain DHCP leases with no reservation. This is a deliberate choice — the long-term fix is decoupling the API endpoint from any node IP via kube-vip (see Architecture Overview) rather than pinning IPs forever. The cost: if a node's lease changes (a reboot during a burst of other hosts also churning leases can cause this even without a subnet change), kubeadm's static pod manifests and several kubeconfigs still point at the old IP and the control plane won't come back on its own.
Full recovery runbook: k8s/dke-mgmt/node-ip-change-runbook.md in the main repo.
Read it in full before improvising a fix — the failure cascades through etcd,
apiserver, kubelet/controller-manager/scheduler kubeconfigs, the Node object's
InternalIP, flannel's public-ip annotation, and even Vault's Kubernetes auth plugin (it
caches a stale client and needs a restart), roughly in that order.
PXE boot
DHCP (Kea, on pfSense) hands out PXE boot options for ESXi installs, and TrueNAS runs the TFTP server that PXE clients fetch from. See DHCP & PXE Boot for the full chain and the boot-menu setup.