DPDK based NIC configuration on servers

Source a DUNE-DAQ environment in order to get access to DPDK tools.

DISCLAIMER for following these instructions: Naturally, the interface names, PCIe device IDs, available NUAM nodes may differ on different systems! Update these to the values that you have on your system!

NIC interfaces lookup

We need to acquire the PCIe device IDs of the Ethernet NICs we are using. To look for available NICs, do:

sudo lspci | grep E810

This results with something similar:

98:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)
98:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)

This Intel NIC has device IDs 98:00.0 and 98:00.1.

Binding the device with the VFIO driver

For using the device with DPDK, we are using the Virtual Function I/O driver. DPDK has tools to do this.

dpdk-devbind.py -s | grep E810

You should see something like this for non-configured NICs:

0000:98:00.0 'Ethernet Controller E810-C for QSFP 1592' if=ens6f0 drv=ice unused=vfio-pci,uio_pci_generic
0000:98:00.1 'Ethernet Controller E810-C for QSFP 1592' if=ens6f1 drv=ice unused=vfio-pci,uio_pci_generic

The interface names are ens6f0 and ens6f1.

Shut down the interface

First, we need to put the interface to DOWN state using the currently used kernel driver (ICE):

sudo ifconfig ens6f0 down
sudo ifconfig ens6f1 down

Load VFIO drivers

sudo modprobe uio_pci_generic
sudo modprobe vfio_pci

Configure hugepages and mount them for DPDK

echo "Add Hugepages on every NUMA node..."
dpdk-hugepages.py -c
sudo echo 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
sudo echo 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
dpdk-hugepages.py -m

Bind the NIC with the VFIO driver

Now that both Hugepages and VFIO are enabled, we can swap drivers:

echo "Bind NIC port with VFIO-PCI instead of ICE"
dpdk-devbind.py -u 0000:98:00.0
dpdk-devbind.py -u 0000:98:00.1
dpdk-devbind.py -b vfio-pci 0000:98:00.0
dpdk-devbind.py -b vfio-pci 0000:98:00.1

Use the "hack" for permissions

For giving absolute permissions for everyone on the server to use DPDK enabled NICs over VFIO, do this:

echo "Nuke permissions for HugeP and VFIO devs..."
sudo chmod -R 777 /dev/hugepages
sudo chmod -R 777 /dev/vfio

Example script

The above steps can be automated via a following script.

REMINDER: Naturally, the interface names, PCIe device IDs, available NUAM nodes may differ on different systems!

#!/bin/bash

#echo "Set MTU for NIC ports..."
ifconfig ens6f1 mtu 9000

ifconfig ens6f1 down


echo "Add Hugepages on every NUMA node..."
dpdk-hugepages.py -c
echo 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
dpdk-hugepages.py -m

echo "Modprobe drivers..."
modprobe uio_pci_generic
modprobe vfio_pci

echo "Bind NIC port with VFIO-PCI..."
dpdk-devbind.py -u 0000:98:00.0
dpdk-devbind.py -u 0000:98:00.1
dpdk-devbind.py -b vfio-pci 0000:98:00.0
dpdk-devbind.py -b vfio-pci 0000:98:00.1

echo "Nuke permissions for HugeP and VFIO devs..."
chmod -R 777 /dev/hugepages
chmod -R 777 /dev/vfio