Skip to content

Terraform

Terraform

Official download

Terraform install/downloads — always current.

Provider and auth

Everything authenticates through Vault-sourced data (data.vault_kv_secret_v2), never .tfvars literals — see Vault Overview for the exact secret paths. provider.tf in each stack wires the vsphere provider credentials from devhome/vmware. See Terraform Providers for the full provider list; the three that shape the stack's overall structure:

  • vsphere (terraform/vsphere/base, compute, supervisor) — the on-prem half. Onboards ESXi hosts into the devhome-cl cluster, wires VLAN 20/30 networking and datastores, and clones VM templates into running VMs — see Compute & Storage for what it actually builds.
  • azurerm (terraform/azure) — the cloud half. Manages the Azure hub resources (VNet, gateways, DNS resolver, private endpoints) described in Cloud: Azure, authenticated from Vault-stored service-principal credentials rather than anything on-prem.
  • vault (terraform/vault) — the odd one out: everywhere else vault is just a read-only secrets source for the other two providers, but this stack is the exception — it's Terraform managing Vault itself (policies, auth backends, and the KV secret engines every other stack reads from), not consuming secrets from it.

Adding a host to the base cluster

Hosts are declared in terraform/vsphere/base/variables.tf's esxi_fqdn map (hostname only) and, if they should carry VLAN 30 workload traffic, in the parallel workload_hosts map (hostname → the pNIC to dedicate as the VDS uplink). Always verify the vmnic mapping against vCenter's Physical Adapters page first — this has bitten more than once when a host's NICs got reseated or renumbered.

null_resource.powercli_init (running bootstrap_esxi.ps1) is wired with replace_triggered_by = [vsphere_host.esxi8], so it automatically re-runs whenever the host set changes — this is what applies iSCSI, NTP, vMotion, and VMFS-snapshot-mount handling to any newly-added host without a separate manual step.

The kube-vip cluster module (modules/k8s)

New workload clusters get a floating BGP-advertised VIP as their controlPlaneEndpoint, decoupling the API endpoint from any single node's IP — Calico already runs its own BGP session to pfSense (ASN 64700) to advertise LoadBalancerIPs, so kube-vip reuses that same precedent with its own distinct ASN (64800) and a second, dedicated source IP on the control-plane node (same NIC/VLAN, no new vNIC), since two independent BGP speakers can't share one source IP with pfSense's FRR. The manifest is hand-templated (matching the repo's existing kubeadm-config.yaml/bgp.yaml sed-substitution convention) rather than generated by kube-vip's own Docker-based generator, since the nodes run containerd.

dke-mgmt does not have this yet — it predates the module and still pins its API endpoint to the node's own DNS-resolved IP, which is exactly why a node IP change is disruptive there (see Kubernetes Troubleshooting). It gets migrated to this design when it's next rebuilt rather than retrofitted live.

Destroying a cluster stack cleanly

Never run a bare terraform destroy against the supervisor/Tanzu-style stack — WCP (Workload Control Plane) state in vCenter's own database gets left behind in ways Terraform can't clean up on its own (orphaned CP VM entities, stale content libraries, a vm_folder_db_configs flag that permanently marks a cluster "namespaces enabled" even after everything else is gone). terraform/vsphere/supervisor/destroy.sh wraps the correct order of operations — pre-destroy VCDB cleanup, terraform destroy -refresh=false, and a post-destroy empty-cluster removal step — and is the only supported way to tear that stack down. If you're touching that script, note that several of its VCDB queries assume a specific VCSA schema version; verify table/column names against information_schema.columns before trusting 2>/dev/null || true error-swallowing to mean "this step is a no-op," since it can also mean "this query is silently broken."

When terraform.tfvars itself is the bug

If a resource is reported "broken" but terraform validate/fmt -check both pass clean, check the consuming .tfvars file for stray non-HCL content before touching the .tf file at all. terraform.tfvars files here are gitignored and hand-edited outside version control, and plain-text scratch notes/TODOs jotted inline (not as # comments) break HCL parsing for the entire module, not just the resource you're looking at — terraform plan -var-file=<real path> surfaces the actual parser error immediately. Confirm with the user before deleting any stray lines found this way — they may be in-progress planning notes, not garbage.

Fixing a ForceNew argument drift without destroying the resource

Some vsphere_virtual_machine arguments (e.g. ovf_deploy.local_ovf_path) are ForceNew on every field — any drift between config and state forces a full destroy+recreate, which is unacceptable for something like the HAProxy VM (whose haproxy.cfg backends/VIPs/certs took real effort to get right and live only on that VM's disk, not in Terraform). terraform import does not help here: local_ovf_path is a one-time deploy instruction, not a live-readable vCenter API attribute — there's nothing for import's Read function to reconcile against.

The actual fix — direct state-file surgery:

  1. Back up first: terraform state pull > backup.tfstate, and separately copy the raw backend file too (belt-and-suspenders in case the backend itself is ever suspect).
  2. Pull state as JSON, find the exact resource instance, edit only the one drifted field to match the new value already in .tfvars, bump serial by 1.
  3. Diff the modified JSON against the original before pushing — confirm only the intended field changed (watch for cosmetic JSON-encoding differences that aren't real changes, e.g. escaped vs. literal characters from different JSON encoders).
  4. terraform state push <modified-file>.
  5. terraform plan should now show "No changes."

This makes zero API calls to vCenter and touches zero real infrastructure — pure Terraform bookkeeping to reconcile state with a config value that changed after the resource already existed. Don't reflexively add ignore_changes to a downstream resource's resulting diff (e.g. a VM-host affinity group's membership) defensively — if it's a genuine side effect of the primary fix, it resolves on its own once the primary field is corrected; masking it would silently break a legitimate future rebuild.

Terraform + Vault backend

State and locking use terraform/vault/* (see packer.tf there for Packer's own Vault-sourced build credentials, mirroring the same pattern as everything else).