System setup for Mellanox (NVIDIA) cards

System preparation

Working with Mellanox (NVIDIA) cards requires the latest driver and device firmware NVIDIA MLNX_OFED.

  1. Using the link in the Download section select the driver for the required distribution kit of the operating system in the table. Tested to work on Debian 10+ and Ubuntu 20.04+.

  2. Download the latest driver (.tgz-archive).

  3. Unpack the archive and start the installation. The installation process will update the firmware of all Mellanox network cards found in the system.

    Installation of the minimum required set of packages:

    ./mlnxofedinstall --basic --without-neohost-backend --without-neohost-sdk

    To update the firmware of network cards without driver installation:

    ./mlnxofedinstall --fw-update-only

    Reboot the system when finished.

    Impoertant: during installation, the OS version check may not occur. In this case, you need to add the --skip-distro-check option.

MITIGATOR setup

The specifics of working with the Mellanox driver is that they remain under Linux control, that is, the network interfaces remain in the system. However, you cannot transfer traffic through them when MITIGATOR is running.

  1. Verify that the output of the dpdk-devbind --status-dev=net command contains the required Mellanox network devices and that they are bound to the mlx5_core driver.

  2. If devices are present, but not bound to the mlx5_core driver, make sure that there is no attempt to bind these devices to another driver (for example, the PCI addresses of devices are NOT found in nics.conf).

  3. Download and install the correct Docker Compose base configuration for Mellanox network cards.

Network card setup

  1. Perform setup of Mellanox network cards firmware with the script mlx_firmware_tune. The script must be run once when installing a new network card. Requires a system reboot to apply changes.

    The script writes performance optimization settings into the card firmware and switches the mode of network ports from InfiniBand to Ethernet if supported.

  2. Apply Mellanox network card settings for performance optimization with the script mlx_runtime_tune. Settings are active during the current session. The script must be run at the system startup by registering it in autoload.

  3. Enable IOMMU pass-through mode. Increases performance (Linux kernel parameter):

    iommu=pt