Skip to content

VMware & Provisioning Overview

VMware

The lab's compute layer is vSphere: a single vCenter (VCSA) managing a cluster of ESXi hosts (devhome-cl), with everything above the hypervisor — hosts, networking, datastores, VMs, and Kubernetes clusters — provisioned by Terraform against the vsphere provider. See Compute & Storage for the cluster's host/VLAN/datastore layout, VCSA Deployment & Post-Deploy Automation below for how the appliance itself gets deployed and rebuilt, and Identity & SCIM for AD/Entra-gated admin access.

The pipeline, end to end

  1. Packer builds golden VM templates from ISOs staged on the NAS — Ubuntu, Windows, and the HAProxy appliance used for Kubernetes control-plane load balancing.
  2. Terraform consumes those templates: it onboards ESXi hosts into the cluster, wires up networking/storage, and clones VMs — including full Kubernetes clusters via kubeadm, orchestrated through a custom module and Ansible provisioners.
  3. Ansible takes over for anything that's ongoing configuration rather than one-time provisioning: cluster start/stop, backups, certificate renewal.
  4. ArgoCD/Rancher take over once a cluster exists — everything that runs inside a cluster is GitOps-delivered, not provisioned by Terraform.

VCSA deployment and post-deploy automation

VCSA is deployed from the host (WSL/macOS) with make deploy-vcsa, not by Terraform — vcsa-deploy's own installer has no Terraform equivalent, so this stays a Makefile target that shells out to it directly:

make deploy-vcsa

runs vcsa-deploy install --accept-eula --acknowledge-ceip <config> inside a remote ado-agent Docker container on truenas-dr.devhome.cloud (over ssh -tt + docker exec), against the VCSA release staged at /mnt/backup/tftpboot/vcsa/vcsa8.0.3 and the deployment spec at config/vmware/vcsa_config.json. Running it inside that container (rather than directly from a dev machine) is what gives vcsa-deploy/ovftool a stable, always-available Linux environment with network line-of-sight to the target ESXi host, instead of depending on whatever's installed locally.

Once the appliance exists, terraform/vsphere/base picks it up via data.vsphere_virtual_machine.vcsa (looked up by var.vcsa_vm_name, depending on vsphere_host.esxi8 since VCSA runs on esxi8b's local disk and can't be discovered until that host is back in inventory) and terraform/vsphere/base/vcsa_post_deploy.tf takes over for everything vcsa-deploy's JSON template has no field for at all — a rebuild doesn't require the multi-hour manual investigation it took to get right the first time:

Resource What a rebuild loses Fix
null_resource.vcsa_ssh_key SSH key trust (fresh OS, no authorized_keys) Pushes the operator's public key into root's authorized_keys and changes root's default shell from the restricted appliancesh to /bin/bash — required for every other SSH-based step below
null_resource.vcsa_root_password_policy Root password expiration reset to the 90-day default Sets max_days_between_password_change: -1 via the appliance's own REST API (/api/appliance/local-accounts/root) — doesn't depend on SSH key trust
null_resource.vcsa_machine_ssl_cert Machine SSL certificate reset to a self-signed VMCA cert Installs the real vcsa.devhome.cloud Let's Encrypt certificate (kept renewed by ansible/ssl-renew-playbook.yml), triggered on the certificate file's content hash (not just VM identity), so a quarterly renewal re-applies automatically without a fresh deploy
null_resource.vcsa_license Fresh 60-day evaluation license Assigns the real license key via govc, triggered on the key's Vault value so a rotation re-applies without a VM replace 
resource "null_resource" "vcsa_machine_ssl_cert" {
  depends_on = [null_resource.vcsa_ssh_key]
  triggers = {
    cert_hash  = filesha256("${local.repo_root}/certs/vcsa.devhome.cloud.cer")
    key_hash   = filesha256("${local.repo_root}/certs/vcsa.devhome.cloud.key")
    chain_hash = filesha256("${local.repo_root}/certs/vcsa.devhome.cloud.ca.cer")
  }
  ...
}

The Machine SSL cert install goes through certificate-manager over SSH, not VCSA's REST API — the REST equivalent has a confirmed bug on this vCenter build. Publishing the issuing CA chain to Trusted Root Certificates first is idempotent by design (the chain is stable across renewals, so a non-zero return from a repeat publish just means it's already trusted, not a failure).

Once VCSA itself is fully bootstrapped, Identity & SCIM covers the separate AD/Entra admin-access layer on top.

Terraform directory layout (terraform/vsphere/)

Directory Owns
base/ The devhome-cl cluster itself, ESXi host onboarding, the shared workload VDS, NFS datastore mounts
compute/ General-purpose VMs (dev VMs, Windows, dke-mgmt)
modules/k8s/ The reusable kubeadm cluster module — used to stand up new dke-* workload clusters
modules/argocd-register-clust/ Registers a cluster into ArgoCD as a target, given a kubeconfig
modules/vault-k8s-auth/ Wires a cluster's ServiceAccount into Vault's Kubernetes auth method
supervisor/ (Retired) the standalone Tanzu Supervisor stack — its cluster/VDS were folded into base/; only HAProxy/content-library/namespace resources remain relevant if this gets rebuilt