Skip to content

Troubleshooting: Networking

Diagnostic sequence for "connection times out" between two lab hosts/containers

  1. Confirm the target address is actually current. DNS records and DHCP leases both drift in this lab — resolve the hostname fresh rather than trusting a previously-known IP.
  2. Raw TCP test first, isolating the network layer from application auth: 
    timeout 5 bash -c 'echo > /dev/tcp/<host>/<port>' && echo OK || echo FAIL
  3. If it hangs (not refused): capture simultaneously on the source's loopback and its physical/external interface while retrying. Answered-but-never-arrives (SYN goes out, SYN-ACK never comes back to the client) points at a routing problem on the server side, not a firewall drop on the client side — check the server's ip rule show / ip route get <dst> from <src> iif <in-iface> before assuming the client's own firewall is at fault. This exact technique found the docker-bridge PBR gap — see TrueNAS Networking Gotchas.
  4. ip route get needs iif to simulate a forwarded packet correctly. Without an iif, the kernel treats the query as a locally-originated packet and can report "unreachable" even when the real (forwarded) packet would route fine — don't trust a bare ip route get dst from src when debugging container/bridge traffic.
  5. Check nftables/iptables-nft together, not just one. Some systems (TrueNAS SCALE included) manage iptables rules via the iptables-nft compat layer — nft list ruleset shows the actual enforced rules and will reveal DOCKER-USER/ DOCKER-FORWARD chains that a plain iptables -L view can miss context on.

DHCP/Kea specifics

See DHCP & PXE Boot for the DNS-registration flag gotcha and Network Design for why Kubernetes nodes specifically have no reservation and what that costs.

Asymmetric routing (dual-NIC hosts)

Full writeup: config/truenas/asymmetric-routing-fix.md in the main repo. Summary and pointer: TrueNAS Networking Gotchas.