Skip to content

Compute & Storage

ESXi cluster (devhome-cl)

All hosts are managed under a single vCenter cluster, devhome-cl, provisioned by Terraform (terraform/vsphere/base). There is deliberately no separate cluster for Kubernetes/Tanzu-style workloads anymore — an earlier design used a standalone Tanzu-POC-Cluster with its own distributed switch, which was retired in favor of folding those hosts into the same base cluster and the same workload VDS.

Host Role Management uplink Workload VDS uplink
esxi8b Hosts VCSA (single NIC) vSwitch0 (only NIC) none — never runs workload VMs
esxi General compute vmnic0 vmnic1 (HPE 530SFP+ port 2, pinned 10G)
esxi-dev General compute vmnic0 vmnic1
trident (MSI H110 Gaming Trident 3) General compute vmnic1 vmnic2
esxi-dev2 General compute vmnic1 vmnic2

Always verify vmnic mapping before applying

Which physical port plays which role is host-specific and has changed at least once per host as NICs got reseated or hosts got rebuilt. Verify against vCenter's Physical Adapters page before any Terraform apply that touches host networking — this is called out directly in terraform/vsphere/base/variables.tf.

Networking: two VLANs, two roles

  • VLAN 20 (Management/Storage) — no dedicated portgroup or storage vmk is managed by Terraform here: each host's UniFi switch port uses the "Management" profile (native VLAN 20, untagged), so vmk0 lands on 10.10.20.x directly via DHCP (Kea reservations keyed to the physical NIC MAC). Storage traffic to TrueNAS rides vmk0, L2-direct — see Network Design for the full VLAN table.
  • VLAN 30 (VM Workloads) — carried by VM-Workloads-vDS, the shared workload switch every general-compute host uplinks into via one dedicated pNIC per host (see the table above). esxi8b deliberately has no workload uplink — it only hosts VCSA.

Storage

  • truenas-nfs — the general-purpose NFS datastore, mounted on every host that needs shared storage (all of them, as of the base-cluster consolidation).
  • vsphere-k8s-gold / truenas-iscsi — iSCSI VMFS datastores backed by TrueNAS LUNs, historically used by the Tanzu/Supervisor stack. iSCSI discovery targets are set via bootstrap_esxi.ps1 (-TruenasIp, sourced from var.truenas_nfs_ip — never hardcoded) against 10.10.20.3 on VLAN 20.
  • vSphere HA needs ≥2 shared datastores per host. A host with only local disk plus one shared datastore will show the "vSphere HA heartbeat datastores... less than required: 2" warning — mount a second shared datastore (usually truenas-nfs) to clear it.

Provisioning flow

  1. PXE boot gets a bare-metal host to an installed ESXi in the first place — Kea hands out PXE options, the host fetches TrueNAS's iPXE menu over TFTP, and picks the ESXi kickstart entry. See DHCP & PXE Boot for the full chain — the same "walk up, pick from a menu, walk away" flow also installs Ubuntu and Windows VMs, not just ESXi hosts.
  2. Packer builds golden VM templates from ISOs staged on the NAS — see Packer.
  3. Terraform (terraform/vsphere/base, compute, supervisor) adds ESXi hosts to the cluster, wires networking/storage, and clones templates into VMs — see Terraform.
  4. bootstrap_esxi.ps1 (a PowerCLI script run as a Terraform local-exec provisioner, triggered whenever the host set changes) handles everything the vsphere Terraform provider doesn't natively support: iSCSI enablement, NTP, vMotion, ATS-heartbeat tuning, and mounting any unresolved "snapshot" VMFS volumes that a freshly-joined host doesn't recognize yet.
  5. Ansible handles ongoing day-2 configuration and lifecycle operations (cluster start/stop, backups, cert renewal) — see Ansible.