Skip to content

Self-Hosted Azure DevOps Agent

Microsoft-hosted ADO agents can't reach anything lab-only — TrueNAS, vCenter, ESXi hosts, Vault. A self-hosted agent solves this by running inside the lab network, as a Docker container on truenas-dr (TrueNAS SCALE Apps, running the ado-agent image, e.g. registry.devhome.cloud/ado-agent:latest).

Why on truenas-dr specifically

truenas-dr is the TrueNAS box that runs Docker/Apps workloads (see TrueNAS Services) and sits on the same VLANs as everything else the pipelines need to reach — no extra hop, no extra VPN client to manage.

Pipelines that depend on it

Pipeline pool: truenas-dr-pool. Pipelines in pipelines/*.yml (backup, ssl-renew, recording-cleanup, terraform, power-state, DNS record management) all target this pool because they run Ansible playbooks or terraform against lab-only inventory.

Build architecture: one script, two consumers

The image (built from docker/Dockerfile in the separate devsetup repo — the same repo that owns physical/VM developer-machine bootstrapping) shares almost all of its tooling install with the WSL host setup script (Developer Laptop Setup's install_dev_tools.sh), rather than duplicating it:

  • install_dev_tools.sh installs everything both WSL and the container need — base packages, git (PPA), Python/Ansible + collections, kubectl, eksctl, cmctl (cert-manager CLI), Helm, k9s, Azure CLI, ArgoCD CLI + Image Updater, Terraform/Vault/Packer, govc. Both make install-ubuntu (WSL) and the Dockerfile (COPY + RUN bash) call this exact same script — one source of truth.
  • Docker engine is the one deliberate exception, container-only: the real WSL host already gets docker via Docker Desktop's WSL integration, so installing a second docker-ce there via apt would conflict with it.
  • configure-personal.sh runs at container start time, not build time, via entrypoint.sh — it needs things that only exist at runtime (the Vault secrets file and NFS-backed config arriving via docker-compose volume mounts, the agent's own SSH key arriving via a runtime-mounted staging path at /mnt/ssh-source, never the Dockerfile). The entrypoint sources this script rather than running it as a subprocess, specifically so export VAULT_ENV_FILE=... survives into the exec'd vault-start.sh that follows — a subprocess's exports vanish the moment it exits; a sourced script's don't.

Private key safety: the user's own personal SSH key (which grants root access to several other lab hosts) is never copied into the image — enforced by both a .dockerignore exclusion and the Dockerfile only COPYing specific named files, never a whole directory, as a second layer of safety.

Build context is the devsetup repo root, not the docker/ subdirectory — required so the Dockerfile can reach the shared install_dev_tools.sh outside docker/. Every COPY path in the Dockerfile is repo-root-relative as a result.

Makefile targets (in the devsetup repo)

make build-ado-agent    # builds registry.devhome.cloud/ado-agent:latest
make push-ado-agent     # pushes it (assumes docker login registry.devhome.cloud)
make deploy-ado-agent   # build + push, then ssh+pull+up on truenas-dr

deploy-ado-agent restarts the live container — only run it when actually ready for that. Deploy itself is just ssh truenas-dr "docker compose -f <path> pull ado-agent && docker compose -f <path> up -d ado-agent" — Docker compares image digests, not the tag string, so re-pulling :latest after a push picks up the new image even though the tag text never changes. The compose file this targets lives permanently at config/truenas/compose/ado-agent/docker-compose.yml (this same devhome repo, NFS- visible from both WSL and via SSH to truenas-dr).

The sibling devhome-memory MCP server (the server backing this documentation site's own persistent-memory system) uses the identical pattern: make build-mcp-server / push-mcp-server / deploy-mcp-server, deploying config/truenas/compose/mcp/docker-compose.yml.

The networking gotcha this container hits

A container on a TrueNAS Apps bridge network trying to reach a service on its own host (e.g. ssh psharma@truenas-dr, which resolves to the host's own IP) is subject to the exact same policy-routing gap documented in TrueNAS Networking Gotchas — sshd answers, but the reply gets source-routed out to pfSense instead of back to the container. If an ADO pipeline job that talks to truenas/truenas-dr over SSH starts timing out after looking fine for a while, check that doc first, specifically the priority-50 docker- bridge exemption rule in pbr-mgmt20.service.

Diagnosing from inside the container

docker exec -it <container-id> /bin/bash
# raw TCP test (bypasses Ansible/SSH auth entirely, isolates network-layer issues)
timeout 5 bash -c 'echo > /dev/tcp/<target-ip>/22' && echo OK || echo FAIL
# reproduce the exact Ansible path a pipeline uses
ANSIBLE_HOST_KEY_CHECKING=false ansible <target-fqdn> -i "<target-fqdn>," -u psharma -b -m ping -vvv

The -vvv ping reproduction is the most reliable way to confirm a fix actually resolves the pipeline's failure, rather than just a raw TCP connection — Ansible's SSH options (ControlPersist, become, Python interpreter discovery) exercise more of the real path than a bare ssh command does.