Reimplementing pf as an eBPF/XDP dataplane on Linux

1. Motivation

I started pfSense in 2004 as a fork of m0n0wall. It is built on FreeBSD, and its firewall is pf — OpenBSD's packet filter, whose configuration language is, in my biased opinion, still the clearest expression of firewall intent anyone has shipped. Two decades of operating that codebase at scale also taught me where its architecture, a product of its era, shows its age: a control plane and a web UI that can drift out of sync, and a dataplane bound to the host kernel's netfilter path.

PFL is not pfSense, and it is not a drop-in replacement for it. It is a narrower experiment with a specific question: can pf's language and stateful semantics be expressed efficiently on Linux's programmable datapath — XDP — rather than on netfilter? The appeal is concrete. XDP runs in the NIC driver (or in hardware-offload mode on capable NICs), before the kernel allocates a socket buffer, and prior work has shown it sustaining packet rates an order of magnitude beyond the conventional path ^[1][9]. The cost of admission is a far stricter programming model, described in §4.

The contribution here is not a new packet-processing primitive — XDP, LPM tries, and BPF conntrack patterns are established. It is the integration: taking a complete, real-world firewall language with state, NAT, normalization, anchors and policy routing, and showing how much of it survives translation into a verifier-accepted XDP program, where it breaks, and what it costs.

2. Background

2.1 The pf configuration language

pf rules read close to intent: block/pass, direction, quick short-circuiting, keep state, address <tables>, macros, anchors, scrub normalization, nat-to/rdr translation, and policy routing via route-to. The language is also a contract with operator muscle memory; a partial imitation that silently drops a clause is worse than none. PFL therefore treats parity — including the awkward corners — as the primary requirement, not a marketing checklist (§5).

2.2 XDP and the eBPF execution model

An XDP program runs on each received frame at the driver level and returns a verdict (PASS, DROP, TX, REDIRECT). It executes only after the in-kernel verifier proves termination and memory safety: bounded loops, a 512-byte stack per program, every pointer access provably in-bounds, and a total complexity ceiling ^[11]. State persists across invocations only in BPF maps. These constraints are the central engineering problem of this project, not an implementation detail (§4).

2.3 Why not the existing options

netfilter/nftables is mature and full-featured but lives past the driver and the sk_buff allocation. bpfilter aimed to JIT iptables rules to BPF but did not land as a general solution. XDP is widely used for stateless DDoS scrubbing and L4 load balancing — Cloudflare's L4Drop, Meta's Katran ^[9], and Cilium ^[10] — but those are purpose-built datapaths, not general stateful firewalls driven by a pf-grade policy language. VPP ^[12] achieves very high rates in userspace at the cost of taking the NIC away from the kernel entirely. PFL occupies a different point: a general pf firewall that stays inside the kernel's datapath and leaves the interface usable by the host.

3. Architecture

PFL has two halves. pfld (~3,700 lines of Rust) parses pf.conf, lowers it to an intermediate representation, and writes the result into BPF maps; it owns no packet path. The dataplane (~7,000 lines) is written in Rust and compiled to eBPF with aya ^[8] — a deliberate choice; the verifier is equally strict regardless of source language, and Rust's type system removes a class of pointer-arithmetic mistakes that are easy to make in C at this layer.

3.1 The pipeline as a tail-call graph

A single XDP program cannot hold a full firewall and stay under the complexity ceiling. The datapath is therefore split into stages that tail-call one another through BPF program arrays (PROG_ARRAY_*), with the matching set duplicated on the TC egress hook for outbound rules:

The FILTER stage is itself 16 tail-called sub-programs (xdp_pfl_0..15), each evaluating a slice of the compiled ruleset; this keeps any single program's instruction count and branch complexity inside the verifier's budget. Stages communicate through an 80-byte scratchpad carried in the packet's data_meta region (via bpf_xdp_adjust_meta), so a later stage never re-derives what an earlier one already computed — addresses, L4 offsets, flags, the conntrack verdict.

3.2 State and the established-flow fast path

Connection state lives in LRU hash maps keyed per address family (FlowStateV4 is 128 bytes; FlowStateV6 is 192). CONNTRACK runs a real TCP state machine (SYN_SENT → SYN_RCVD → ESTABLISHED → … → TIME_WAIT), not a binary new/established flag. An in-state packet takes the short path: one map lookup, an in-place state mutation via a mutable map-value pointer (no full value rewrite), an incremental checksum update per RFC 1624 ^[3] rather than a recomputation, and a forwarding decision served from a FIB/L2 result cached in the flow state on the first packet — so the steady-state cost excludes a routing-table walk. The packet path takes no atomics; per-CPU maps hold all counters and diagnostics.

The same pf.conf an existing operator already knows compiles unchanged:

# normalization, state, NAT, overload tables — standard pf
set skip on lo
scrub in all max-mss 1440 random-id
nat on egress from 10.0.0.0/8 to any -> (egress)
block in quick from <bruteforce>
pass in on egress proto tcp to port 443 keep state \
     (max-src-conn 100, max-src-conn-rate 15/5, overload <bruteforce>)

4. Implementing under the verifier

The bulk of the engineering effort was not writing firewall logic; it was making correct firewall logic that the verifier will accept. Three constraints dominated.

Stack budget (512 bytes). A full pf rule evaluation, plus NAT translation, plus the IPv6 paths, does not fit in one stack frame. The response is structural: cold and stack-heavy operations live in their own programs, reached by tail call, each with a fresh frame. NAT was split into a dedicated program when it overflowed; the IPv6 NAT/redirect chains overflowed again and were given a separate program (xdp_pfl_natv6) rather than sharing the v4 frame. REJECT, NAT64 and the SYN-cookie responder are likewise isolated.

Bounded execution. The verifier rejects unbounded loops. Rule evaluation iterates over a compile-time-bounded slice per FILTER chunk; ruleset growth is absorbed by adding chunks (tail calls), not by longer loops. This is why FILTER is 16 programs rather than one loop over N rules.

Toolchain. Building a Rust eBPF object reproducibly on a musl host required pinning the workspace to an LLVM-21 nightly matching the BPF linker, so the dataplane builds in the same CI as the rest of the system rather than on a bespoke machine. All 25 programs load and attach in native driver mode on Linux 7.0.

Writing the datapath in Rust did not relax any verifier rule, but it did make the failures legible: bounds and lifetimes are expressed in the type system, and what remains is a genuine verifier disagreement about stack or complexity rather than an accidental out-of-bounds access. The verifier log, read like a stack trace, points at the offending program; the fix is almost always to split it.

5. pf feature parity

Each implemented feature landed with an automated test — a parser unit test, a veth forwarding test, or both. Table 2 summarizes the current state by category; the honest column is the last one.

What XDP cannot do, stated plainly. Out-of-order fragment reassembly is not implemented and will not be in the fast path. An XDP program sees one frame per invocation and cannot buffer and reorder fragments across invocations; reassembly requires holding state and bytes that the model does not provide. PFL forwards non-first fragments with a stored next-hop and drops what policy requires, but it does not reconstruct datagrams. Where a feature is precluded, the parity document records "impossible here, and why" rather than implying coverage.

6. DoS resistance and security

SYN cookies in XDP. When a synproxy rule matches, a SYN flood is answered with a cryptographic SYN-ACK generated in the driver, and no connection state is created until a returning ACK validates the cookie. The cookie ISN uses Half-SipHash-1-3 ^[6] — the same construction and parameters as the Linux kernel's own SYN-cookie path (hsiphash) ^[7], with a dual-slot secret for rotation. Because state is created only post-validation, a spoofed-source flood cannot exhaust the state table; a stress run of 5×10⁵ spoofed SYNs left it untouched.

Fail-closed. If pfld dies, the firewall must not vanish and leave the interface open. The XDP and TC programs are pinned into the BPF filesystem, so they remain attached and enforcing across a daemon crash — the failure mode is closed, not open. This depends on bpffs program pinning and targets Linux ≥ 7.0.

Review. Before exposure to testers, the datapath and control plane were audited; 66 findings (40 system-wide, 26 PFL-specific) — spanning a TCP-injection window, cookie-key handling, and an authenticated command-injection path — were resolved. The audit also drove dataplane changes reflected above (e.g. per-CPU rule counters replacing a shared array). I make no claim that this constitutes a security guarantee; it is the diligence done to date, on an experimental component.

7. Evaluation

This is where I will under-deliver relative to a launch post, on purpose. The only controlled measurement to date is on a 2-vCPU CI virtual machine comparing PFL against the same host's nftables backend on identical rules and traffic.

PFL wins median latency and currently loses tail latency and CPU efficiency. The cause is identified, not mysterious: forwarded packets re-enter the FILTER stage to apply out rules, and that re-entry re-parses and re-evaluates even for established flows. Caching the egress verdict per flow — exactly as the FIB result is already cached — is the in-progress fix, and the per-CPU-counter change above is part of the same effort.

7.1 Threats to validity

A virtual machine is the wrong instrument for a firewall throughput claim. Hypervisor virtio paths, host scheduling, and a shared memory bus make any large packets-per-second or Gbps figure an artifact of the host, not a property of the firewall — so I am not quoting one. The latency comparison above is on a like-for-like VM and is informative as a relative result between two backends on the same host; it is not a wire-speed number. XDP's order-of-magnitude advantage over the netfilter path is established in the literature on real hardware ^[1][9]; whether PFL realizes that under a full pf ruleset is precisely the open question §8 commits to answering on bare metal.

8. Limitations and future work

• Bare-metal evaluation. The next measurement is on Arm appliances (Marvell OCTEON 10) whose NIC driver supports native XDP, to report on-silicon latency and pps under realistic rulesets — the numbers the VM cannot provide. Until then, treat all performance statements as preliminary.
• Egress fast path. The forwarded-packet re-entry cost (§7) is the headline performance work remaining.
• High availability. A pfsync analog (pflsync, HMAC-authenticated state replication) exists but is early and gated; production two-node failover is unfinished.
• Protocol depth. modulate state, richer normalization, and the genuinely hard edge cases of fragment handling remain.
• Maturity. PFL is opt-in per interface and experimental. The conservative, well-trodden path remains nftables; PFL is offered as the faster path for those who want it, not as a default.

9. Related work

XDP itself ^[1] is the foundation. Production XDP datapaths — Katran ^[9], Cilium ^[10], Cloudflare's L4Drop — demonstrate the performance envelope but target load balancing and stateless scrubbing rather than a general policy-driven stateful firewall. VPP ^[12] reaches higher rates by leaving the kernel datapath entirely. nftables remains the reference for in-kernel stateful filtering breadth, and bpfilter was an earlier attempt to bridge iptables semantics to BPF. PFL's distinguishing aim is fidelity to the pf language and semantics on the in-kernel XDP datapath, with the host interface left intact.

10. Conclusion

Most of pf — its language, its stateful core, NAT including NAT64, normalization, anchors and policy routing — can be expressed as a verifier-accepted XDP dataplane, provided the program is structured around the verifier's constraints rather than against them: tail-called stages, isolated stack-heavy paths, bounded rule evaluation, and a scratchpad to avoid recomputation. Some pf behavior (full fragment reassembly) is genuinely outside the model, and the performance case is, honestly, half-made: a median-latency win in a VM, a known tail-latency cost, and a bare-metal evaluation still owed. I would rather state that plainly than oversell it. The work, the parity tracker, and the eventual hardware numbers will be published as they land.