Secrets Engines

Vault supports multiple kinds of secrets engine side by side. devhome uses three, and they represent two fundamentally different models — worth understanding before adding a fourth:

KV-v2 (devhome/) — static storage. You put a value in, you get the same value back out, forever, until something explicitly overwrites it.
Azure (azure/) and Kubernetes (kubernetes-dke-mgmt/) — dynamic secrets. Vault actively talks to an external system (Azure AD, a cluster's API server) and generates a brand-new, short-lived credential on every request, then tears it down again when the lease expires.

See Vault Overview for the KV path layout and Kubernetes Auth for the (separate, easily confused) Kubernetes auth method — that's pods authenticating to Vault; this page is about Vault generating credentials for other systems.

`devhome` — KV-v2 (static)

Not created by Terraform at all — the mount itself predates this repo's Terraform-managed state and is assumed to already exist. What Terraform manages is the data inside it: one vault_kv_secret_v2 resource per logical secret group (terraform/vault/vmware.tf, azure.tf, argocd.tf, rancher.tf, packer.tf, netbox.tf, dir_path.tf), each writing a data_json blob built from variables sourced from the gitignored terraform.tfvars. Simple, and the right tool for anything that doesn't need to rotate itself — which is everything else in this repo except the two engines below.

`azure/` — Azure secrets engine (dynamic)

terraform/vault/azure-secrets-engine.tf:

resource "vault_azure_secret_backend" "azure" {
  path            = "azure"
  subscription_id = var.arm_subscription_id
  tenant_id       = var.arm_tenant_id
  client_id       = var.arm_client_id
  client_secret   = var.arm_client_secret
  environment     = "AzurePublicCloud"
}

resource "vault_azure_secret_backend_role" "reader" {
  backend = vault_azure_secret_backend.azure.path
  role    = "reader"
  azure_roles {
    role_name = "Reader"
    scope     = "/subscriptions/${var.arm_subscription_id}/resourceGroups/${var.resource_group}"
  }
  ttl     = 3600
  max_ttl = 86400
}

The root config reuses the same ARM service principal already stored statically at devhome/azure (see Vault Overview) — no separate bootstrap credential. That SP needs enough Azure RBAC to create new service principals and assign roles on its own behalf (typically Owner or User Access Administrator + Application Administrator in Entra ID).

vault read azure/creds/reader

mints a brand-new service principal, scoped to Reader on var.resource_group, and deletes it from Azure AD automatically when the lease (ttl) expires.

Azure AD replication lag

A freshly-minted service principal's first real use can occasionally fail auth — Azure AD hasn't finished replicating it yet. Retry after a few seconds rather than assuming the role/root config is broken.

`kubernetes-dke-mgmt/` — Kubernetes secrets engine (dynamic)

Two ways this gets created exist in the repo right now, and they will conflict with each other if both are applied — see the warning below before running terraform apply on either stack.

The module (the intended, automated path)

terraform/vsphere/modules/vault-k8s-auth — the same module that already wires up the Kubernetes auth method (ESO's Vault login) for every cluster — got an opt-in enable_secrets_engine flag so the secrets engine can be automated per-cluster the same way, instead of being hand-written once per cluster:

module "vault_auth_dke_mgmt" {
  source = "../modules/vault-k8s-auth"
  # ...existing auth-method args unchanged...
  enable_secrets_engine     = true
  secrets_engine_mount_path = "kubernetes-dke-mgmt"
}

When enabled, the module does three things a caller doesn't have to think about:

Creates a separate vault-secrets-engine ServiceAccount bound to cluster-admin — deliberately not reusing the auth method's vault-auth SA, which only has system:auth-delegator (enough to validate tokens, nowhere near enough to create new ServiceAccounts/RoleBindings on the fly, which the secrets engine needs to do for every lease it issues).
Mounts and configures the Kubernetes secrets engine via the vault CLI (not a native vault_kubernetes_secret_backend resource) — same reasoning as the auth method's config step: cluster credentials (host/CA/token) can only be read at apply time, never during plan/refresh, since the cluster doesn't exist yet on a first apply.
Creates a default viewer role: read-only (view built-in ClusterRole), scoped to var.secrets_engine_default_role_namespaces (default ["default"]").

Add enable_secrets_engine = true to any of the module's other callers (dke, dkes, rke2, tkc-dev) to get the same thing for that cluster.

The standalone version (currently live for `dke-mgmt`)

terraform/vault/kubernetes-secrets-engine.tf predates the module flag — it declares the same mount + role directly in the vault stack, pointed at dke-mgmt via three manually-populated variables (dke_mgmt_kubernetes_host, dke_mgmt_kubernetes_ca_cert, dke_mgmt_vault_sa_jwt — the last one is a long-lived bootstrap token, created by hand against the cluster; see the comment above those variables in terraform/vault/variables.tf for the exact kubectl bootstrap commands).

These two will collide if both get applied

dke_mgmt.tf's module call and kubernetes-secrets-engine.tf both try to own the same Vault path (kubernetes-dke-mgmt/), from two different Terraform state files (vsphere/compute and vault). The standalone version is what's actually live right now. Delete terraform/vault/kubernetes-secrets-engine.tf before the next terraform apply on vsphere/compute picks up the module's enable_secrets_engine = true — otherwise that apply will fail trying to mount a path that already exists.

Using it

# NOT `vault read` - this engine's creds endpoint requires a write (POST), since a
# role can span multiple allowed namespaces and Vault needs to know which one you want.
vault write kubernetes-dke-mgmt/creds/viewer kubernetes_namespace=default

returns a service_account_token for a ServiceAccount Vault just created in default, bound read-only to the view ClusterRole. vault lease revoke <lease_id> deletes that ServiceAccount immediately; otherwise it's cleaned up automatically when the lease TTL expires.

Testing with a token: don't reuse an admin kubeconfig's context

kubectl --token=<vault-issued-token> does not reliably override an existing kubeconfig context that already has a client certificate embedded in it — some clients send both, and the API server's certificate authenticator can win, silently running your command as whatever admin identity the certificate represents instead of the scoped token you meant to test. Build an isolated kubeconfig containing only the cluster's server/CA and the token before testing a role's boundaries against a live cluster — verified the hard way, not hypothetically.

Not a good fit for anything that needs to stay connected continuously (ArgoCD, Rancher) — those keep using the auth method (standing identity) instead; see Kubernetes Auth.

Secrets Engines

devhome — KV-v2 (static)

azure/ — Azure secrets engine (dynamic)

kubernetes-dke-mgmt/ — Kubernetes secrets engine (dynamic)