Skip to content

pfSense Overview

pfSense

pfSense is the router/firewall for the entire lab, running in WAN bridge mode behind the ISP modem. It owns:

  • Routing and inter-VLAN firewall rules (LAN/Clients, VLAN 20 Management/Storage, VLAN 30 VM Workloads, VLAN 200 ILO out-of-band management)
  • DHCP (Kea) — see DHCP & PXE Boot
  • DNS resolution and split-horizon overrides for Azure — see DNS
  • Remote access VPN (OpenVPN) and site-to-site connectivity (IPsec) — see VPN
  • Public TLS certificates via ACME against an Azure-hosted public DNS zone — see ACME Certificates
  • Outbound filtering / threat-list blocking via pfBlockerNG — see pfBlockerNG

Official download

pfSense CE downloads — always shows the current release; check the changelog before upgrading a router that's this central to the lab.

Kea, not ISC dhcpd

pfSense migrated its DHCP server from the legacy ISC dhcpd to Kea as part of the network redesign. This wasn't cosmetic — several settings moved to different places in the UI (notably DHCP-lease DNS registration, now gated by a dnsreg flag rather than the old DNS Resolver checkbox), and Kea's lease-handling behavior differs enough from ISC dhcpd that it's a real suspect whenever something DHCP-adjacent misbehaves that "used to just work." See DHCP & PXE Boot for the specific gotchas hit so far.

Hardware history: why routed inter-VLAN transfers used to fail

Before the current build, the production pfSense box's LAN/VLAN trunk NIC (oce1, an Emulex OneConnect BE3 chip) had a confirmed multi-queue/RSS bug: any transfer routed between two VLAN sub-interfaces sharing that one physical NIC (e.g. LAN↔VLAN30) reliably died at a fixed ~15-20 second mark with Input/output error/Connection reset by peer, while same-subnet transfers were completely unaffected. Caught live by polling pfctl -s state: the connection's pf state got stuck in SYN_SENT:CLOSED for the entire failing window even though real bytes were actively transferring, then flipped to TIME_WAIT the instant the client saw the reset — pf's own state machine never saw the handshake complete, because the VLAN tag insert/remove step interacted with the NIC's RSS hash and sent a flow's outbound SYN and return SYN-ACK to different CPU cores' queues. This is specific to two VLAN sub-interfaces sharing one physical multi-queue NIC — it never affected LAN↔WAN traffic (genuinely separate physical NICs), which is why everyday internet access never showed the symptom.

Fix: replaced the Emulex oce1 card with an Intel X520 (82599 chip, ixgbe driver) — the same chip family already proven solid on the esxi host. Intel's RSS implementation uses a symmetric Toeplitz hash, keeping forward/reverse packets of one flow on the same core, which is the exact property the Emulex driver got wrong. This is why the current interfaces are named ix0/ix0.20/ix0.30 rather than oce1/etc.

Practical implication that outlived the hardware fix: since TrueNAS didn't strictly need a dedicated VLAN 30 presence once the NIC bug was fixed, but giving it one sidesteps any future routing weirdness entirely for the highest-value traffic pattern (bulk VM storage access), the interface plan gives TrueNAS a direct tagged VLAN 30 interface anyway rather than relying on routing through pfSense for it.

Restoring a config with VLAN-tagged interfaces

Restoring a config.xml where LAN/OPT interfaces are assigned to 802.1Q VLAN sub-interfaces does not correctly compile the firewall ruleset against those VLAN interfaces on pfSense 2.8.x — pfctl -sr shows every rule (including the anti-lockout rule) still bound to the raw parent NIC instead of the actual VLAN sub-interface, even though config.xml on disk is correct. Symptom: DHCP works (pfSense exempts it from filtering), but ping/HTTPS/SSH to the interface's own IP all silently time out. A full reboot does not fix it. Fix: open each affected interface's settings page (Interfaces → LAN, → OPT1, etc.) and click Save with no changes — this forces pfSense to recompile/re-register the interface for filtering. Verify with pfctl -sr | grep -i <interface> showing the real VLAN ifname, not the parent, before trusting any anti-lockout/default-allow rule after a config restore.

BGP peering

pfSense runs FRR and peers via BGP with Calico (ASN 64700) to advertise LoadBalancerIPs for in-cluster Services, and with kube-vip (ASN 64800, a distinct ASN since FRR peer-groups can only carry one remote-as each) for the floating control-plane VIP on kube-vip-enabled clusters. The peer-group/description prefix is parameterized in terraform/vsphere/compute/pfsense-bgp-playbook.yml specifically so the same playbook can be invoked twice — once per peer — without duplication.