Aeron DPDK - Data Plane Development Kit

Aeron Transport Kernel Bypass is designed to unlock the full potential of your network by leveraging kernel bypass technologies. These technologies allow direct access to the TX/RX ring buffers of network cards, providing APIs designed for performance with zero copy semantics and real-time lock-freedom/wait-freedom. This significantly reduces CPU overhead and yields lower and more predictable latency.

The Data Plane Development Kit (DPDK) is part of the Linux Foundation. Aeron provides support for DPDK with AWS Nitro cards.

How to get DPDK

DPDK is a Premium Aeron component. You can find pre-compiled bundles of the C Media Driver that include DPDK for popular Linux distributions under aeron-dpdk in the Adaptive Artifactory.

Current restrictions/limitations

  • IPv4 Only (no IPv6 support)

  • Unicast Only (no Multicast support)

  • Tested using the following PMD drivers:

    • net_ena (AWS Nitro network interface card)

    • net_ixgbe (Intel network interface card)

  • Single DPDK port/interface per driver

  • Installs as an executable aeronmd_dpdk, which is run instead of aeronmd.

  • DPDK requires that CPU affinity is set.
    Without it all threads get pinned to the same core resulting in poor performance.

DPDK specific configuration options.

  • AERON_DPDK_USER When running as a root level user, after DPDK setup has completed switch to this user.

  • AERON_DPDK_GROUP When running as a root level user, after DPDK setup has completed switch to this user.

  • AERON_DPDK_LOCAL_IPV4_ADDRESS IP address of the local interface.

  • AERON_DPDK_GATEWAY_IPV4_ADDRESS IP address of the subnet’s gateway.

Setup on AWS

It is possible to run the aeronmd_dpdk binary as a non-privileged user, however it requires support from the IOMMU. Support for this in virtualised environments seems scarce currently, especially in AWS. So for AWS the recommended approach is to launch the process as a privileged user (perhaps using sudo) and use AERON_DPDK_USER and AERON_DPDK_GROUP to switch to the appropriate user and AERON_DIR to explicitly set the aeron directory.

Quick Setup

  1. Setup two ENA capable Linux instances (c5n.2xlarge work well). Ubuntu 20.04 and Rocky Linux 8 have been tested.

    1. You will need to ensure that Python 3 and libnuma are installed (generally available from the OS’s package manager)

  2. Provision two additional network interfaces and attach one each to the two instances created in step 1.

  3. Create and bind IP addresses to those new interfaces. One interface will be for normal traffic and the other will be exclusively for Aeron DPDK data. In order for networking to work correctly the Aeron DPDK driver needs to know the ip address of the default gateway for your local subnet.

  4. Update the security group so that the interfaces will accept all UDP traffic from other hosts in the same security group. Ensure that all the interfaces have the same security group.

  5. Log into the instances and run ip a. There should be at least two interfaces listed (excluding loopback). Probably something like ens5 & ens6 or eth0 and eth1.

  6. Run ip route and make note of the default gateway that will be used for the interface that will be used for DPDK.

  7. Disable the interface to be used for Aeron DPDK, e.g. for ens6, use sudo ip link set ens6 down.

  8. Create a directory and untar the Aeron DPDK package file into it.

  9. From within the scripts subdirectory run the ./setup-aws.sh script.

  10. Launch the aeronmd_dpdk binary to start the media driver. The -l 0 disables the DPDK libraries custom threading. The --preserve-env allows for the environment variables used for configuration to be passed through. It is recommended that you launch as the desired non-privileged user and use sudo to get the necessary permissions to initialise DPDK

    > export AERON_DPDK_USER=${USER}
    > export AERON_DPDK_GROUP=$(id -g -n ${USER})
    > export AERON_DIR=/dev/shm/aeron-${USER}
    > export AERON_DPDK_GATEWAY_IPV4_ADDRESS=<gateway ip from step 5>
    > sudo --preserve-env ./aeronmd_dpdk -l 0

Setup (Non-AWS and more complicated configurations)

For more complicated setups or a non-AWS system this section will go through each of the parts of the necessary system configuration and what the setup should be for a root and non-root usage.

This assumes the operations are run as a non-root user and will use sudo when root level access is required.

Huge Pages

DPDK uses huge pages to map memory to allow DMA from the various PMD devices. There are configuration approaches that can be used without huge pages, but going to ignore those as they are likely less useful in production setups. Most modern Linux systems will already have a mount setup for huge pages, typically /dev/hugepages, check for this using:

$ mount | grep huge
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)

If it is not setup for your system, lookup the appropriate documentation for your system in order to get it configured. To set up huge pages for Aeron DPDK use the following command:

$ echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
1024

This will allow for up to 1024 2MB huge pages to be allocated on the system. For a total of 2GB, this should be sufficient for the DPDK bindings. If you are using huge pages for the aeron term buffers as well, you will need to allocate more.

Non-Root Huge Pages

When running as a non-root user you will need to set the necessary privileges on the /dev/hugepages file system.

$ sudo chgrp <group> /dev/hugepages
$ sudo chmod 0775 /dev/hugepages

For <group> chose one that the user you want to run DPDK as is already a member of or create a new group and add the user to it.

Set IOMMU settings

The IOMMU is a feature of the kernel that allows virtualisation of I/O memory addresses in a similar manner to regular memory addresses. In order to use DPDK as a non-root user, we need to make use of it to get the necessary device addresses to access the underlying driver.

IOMMU running as root

When running as root, we will disable IOMMU access:

echo 1 | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

IOMMU running as non-root

Firstly the kernel boot configuration options need to be updated to enable the IOMMU. Ensure the follow settings are included in the kernel boot parameters:

iommu=on intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1

Disable the unsafe mode.

echo 0 | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

Device and Driver Configuration

Firstly ensure that the vfio-pci driver module is installed into the kernel. It is also possible to use the uio driver, however this is an older driver and Aeron DPDK hasn’t been tested with it.

sudo modprobe vfio-pci

Use the dpdk-devbind.py script to list and change the drivers that a device in bound to in order to make it available for Aeron DPDK. Ensure the python3 is installed.

Show the available devices

$ ./dpdk-devbind.py --status-dev net

Network devices using kernel driver
===================================
0000:01:00.0 'MT27800 Family [ConnectX-5] 1017' if=enp1s0f0 drv=mlx5_core unused=vfio-pci *Active*
0000:01:00.1 'MT27800 Family [ConnectX-5] 1017' if=enp1s0f1 drv=mlx5_core unused=vfio-pci
0000:43:00.0 'I211 Gigabit Network Connection 1539' if=enp67s0 drv=igb unused=vfio-pci *Active*
0000:44:00.0 'Wi-Fi 6 AX200 2723' if=wlo2 drv=iwlwifi unused=vfio-pci
0000:48:00.0 '82598EB 10-Gigabit AF Dual Port Network Connection 10f1' if=enp72s0f0 drv=ixgbe unused=vfio-pci
0000:48:00.1 '82598EB 10-Gigabit AF Dual Port Network Connection 10f1' if=enp72s0f1 drv=ixgbe unused=vfio-pci

The values in the left-hand column are the PCI addresses of the devices available (e.g. 0000:48:00.0). Towards the right the interface name is shown (if=enp72s0f1). The name of the driver that is currently bound to the device and the available one are shown (e.g. drv=ixgbe unused=vfio-pci)

Before changing the binding for a device the device should not have a ``UP'' interface locally:

$ ip link show enp72s0f0
5: enp72s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:55:3b:93 brd ff:ff:ff:ff:ff:ff

Set the link down to allow it to be used by DPDK

$ sudo ip link set enp72s0f0 down

Unbind the device from the kernel driver, in the case of Intel this is the ixgbe driver. Then bind it to the vfio-pci driver. Note that the device needs to be referenced by its PCI address.

$ sudo ./dpdk-devbind.py -u 0000:48:00.0
$ sudo ./dpdk-devbind.py -b vfio-pci 0000:48:00.0
$ ./dpdk-devbind.py --status-dev net

Network devices using DPDK-compatible driver
============================================
0000:48:00.0 '82598EB 10-Gigabit AF Dual Port Network Connection 10f1' drv=vfio-pci unused=ixgbe

This makes the device available for use within DPDK.

Non-root Device and Driver Configuration

To allow the non-root user to see the device the permissions on the device need to be updated to allow access to the desired user. To figure out which device file needs to be updated, first we need to know the IOMMU group of the appropriate network card. For this we need the PCI address of the device that has been bound to the vfio-pci driver.

$ readlink '/sys/bus/pci/devices/0000:48:00.0/iommu_group'
../../../../kernel/iommu_groups/52

This device is in group 52 which should be available as a vfio device file:

$ ls -l /dev/vfio/
total 0
crw------- 1 root root 241,   0 Mar  8 15:45 52
crw-rw-rw- 1 root root  10, 196 Mar  8 06:51 vfio

In a similar fashion to setting up non-root access to /dev/hugepages setup the appropriate group and permissions

$ sudo chgrp mike /dev/vfio/52
$ sudo chmod 0660 /dev/vfio/52