router: Add an underlay implementation that uses AF_XDP.

[jiceathome]

This task aims at obtaining the best network IO performance we can for the router while keeping with the mainstream Linux APIs.

AF_XDP admits several usage mode, depending on driver capabilities; one of which named zero-copy. We shall attempt to use that if available. This is not necessarily as fast as DPDK, which can map the entire device in use space.

The option of adding a DPDK implementation was considered and rejected for the following reasons:

• XDP is improving and catching up with DPDK, making DPDK obsolete and the efforts wasted.
• DPDK creates complicated build constraints in terms of available libraries, platforms and devices.
• Once the router as support for multiple underlay implementations, a vendor desiring to specialize a SCION implementation for a subset of devices via DPDK can implement a new DPDK underlay. It allows vendors to have some proprietary value-add if they so chose.

[romshark]

I found a working and pretty well performing AF_XDP Go wrapper
https://github.com/slavc/xdp/blob/main/examples/sendudp/sendudp.go

ubuntu:~/ringtest/send_goxdp$ go build -o snd . && sudo ./snd -i ens5 -dstip $IP
sending UDP packets from 192.168.111.10 (b2:96:81:75:b2:11)...
949518 packets/s (11500 Mb/s)
949113 packets/s (11495 Mb/s)
944044 packets/s (11434 Mb/s)
953412 packets/s (11547 Mb/s)
933518 packets/s (11306 Mb/s)

These results show ~11.5gbit/s ~950k PPS single-core on an AWS EC2 c6in.2xlarge instance
Which is significantly (>2x) faster than the AF_INET io_uring implementation in C that maxed out at ~4.8gbit/s with ~400k PPS (MTU 1500).
Even a c6i.large, a considerably less powerful instance still shows ~1M PPS:

948281 packets/s (11485 Mb/s)
1001033 packets/s (12124 Mb/s)
1017073 packets/s (12318 Mb/s)
1024835 packets/s (12412 Mb/s)
1026498 packets/s (12432 Mb/s)

The only "problem" so far is that this needs sudo privileges, otherwise it seems it can't attach itself to the NIC.

P.S. An io_uring based batched sender only resulted in lower CPU usage due to less syscall overhead, but didn't have any influence on PPS/core.

[jiceathome]

Looks like a promising foundation. Regarding sudo; don’t worry about it. Raw sockets already need it (that or a bunch of capabilities: CAP_NETADMIN, CAP_RAW, and at least one other - that’s what we use for the new underlay. The packaging and the tests of the router take care of it). I would strongly suggest merging the new undelay code as it is first, though; before it starts bitrotting). Til’ has made a couple of valid suggestions regarding comments (mostly). I’ll take care of those in the next few days. J-COn 12 Sep 2025, at 15:32, Roman Sharkov ***@***.***> wrote:romshark left a comment (scionproto/scion#4734) I found a working and pretty well performing AF_XDP Go wrapper https://github.com/slavc/xdp/blob/main/examples/sendudp/sendudp.go ubuntu:~/ringtest/send_goxdp$ go build -o snd . && sudo ./snd -i ens5 -dstip $IP sending UDP packets from 192.168.111.10 (b2:96:81:75:b2:11)... 949518 packets/s (11500 Mb/s) 949113 packets/s (11495 Mb/s) 944044 packets/s (11434 Mb/s) 953412 packets/s (11547 Mb/s) 933518 packets/s (11306 Mb/s) These results show ~11.5gbit/s ~950k PPS single-core on an AWS EC2 c6in.2xlarge instance Which is significantly (>2x) faster than the AF_INET io_uring implementation in C that maxed out at ~4.8gbit/s with ~400k PPS (MTU 1500). Even an c6i.large, a considerably less powerful instance still shows ~1M PPS: 948281 packets/s (11485 Mb/s) 1001033 packets/s (12124 Mb/s) 1017073 packets/s (12318 Mb/s) 1024835 packets/s (12412 Mb/s) 1026498 packets/s (12432 Mb/s) The only "problem" so far is that this needs sudo privileges, otherwise it seems it can't attach itself to the NIC. P.S. An io_uring based batched sender only resulted in lower CPU usage due to less syscall overhead, but didn't have any influence on PPS/core. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[romshark]

I've finished building the 25 gbit/s BlueField-2 workstation which is fully AF_XDP capable, including zero-copy mode.

I wrote this benchmark in C https://gist.github.com/romshark/3dddcf65d877fdf3bb78f434132a9ea5
which demonstrated following over-the-wire results between two FP28-connected BlueField-2 DPUs:

mode	MTU	packets/s	bandwidth
XDP_COPY	1360b	1,215,904	13,229 Mbit/s
XDP_ZEROCOPY	1360b	2,258,397	24,571 Mbit/s

In zero-copy mode, they practically run at line rate (25G) with one CPU core utilized at 100%.
This is just a single-queue, single-threaded sender.

[jiceathome]

Nicely done!Your bandwidth definition may be optimistic though. You measure the best one second ever; which might not be sustainable. Do you get the same results over longer time windows?What’s that rig you got?J-COn 25 Nov 2025, at 18:00, Roman Sharkov ***@***.***> wrote:romshark left a comment (scionproto/scion#4734) I've finished building the 25 gbit/s BlueField-2 workstation which is fully AF_XDP capable, including zero-copy mode. I wrote this benchmark in C https://gist.github.com/romshark/3dddcf65d877fdf3bb78f434132a9ea5 which demonstrated following over-the-wire results between two FP28-connected BlueField-2 DPUs: mode MTU packets/s bandwidth XDP_COPY 1360b 1,215,904 13,229 Mbit/s XDP_ZEROCOPY 1360b 2,258,397 24,571 Mbit/s In zero-copy mode, they practically run at line rate (25G). This is just a single-queue, single-threaded sender. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[romshark]

Nicely done!Your bandwidth definition may be optimistic though. You measure the best one second ever; which might not be sustainable.

Thanks! I've now ported the sender to pure Go (no cgo): https://github.com/romshark/afxdp-bench-go. The hardware specs are listed there. I just ran the zero-copy (MTU 1360b) benchmark for 18m36s sending 2.5b packets (2,517,810,287 = ~3TB) over the wire and I couldn't observe any dips below ~24,313 Mbit/s. In fact, it ran consistently around ~24,570 Mbit/s.

Copy-mode runs at ~10 gbps while the C equivalent ran at 13 gbps. I'm sure this can be improved upon.

[romshark]

The sockets can now do both sending and receiving: romshark/afxdp-bench-go@73ec26b in pure Go. Copy-mode is now also on par with the C version at ~13 gbps.

[romshark]

https://gist.github.com/romshark/91b75b2b5e414b2c068fcf468570d997

I tried to use XDP_SHARED_UMEM (see UMEM) but I couldn't get it to bind sockets across NICs.

This line likely lead to my confusion:

It works on the same queue id, between queue ids and between netdevs/devices.

But it also states:

An UMEM is associated to a netdev and a specific queue id of that netdev.

It seems my initial thought that I could attach an UMEM not just to multiple sockets, but multiple sockets across different devices and queues and then move packets around without having to copy them between UMEMs was wrong. It seems that XDP_SHARED_UMEM is meant to shared UMEM between AF_XDP sockets on the same device:queue combination. This means I can do zero-copy RX/TX, but I'd still have to copy the packet from one NIC socket to another via CPU.