System Setup for Mellanox (NVIDIA) Adapters

System preparation

Working with Mellanox (NVIDIA) cards requires the latest driver and device firmware NVIDIA MLNX_OFED.

1. Using the provided link select the latest version of MLNX_OFED for the required distribution of the operating system in the table Current Versions of the Download section. Select the closest version of the OS if the required version is missing. Skip MLNX_OFED installation if the required OS is missing in the list. Tested to work on Debian 10+ and Ubuntu 20.04+.

2. Download the .tgz archive of MLNX_OFED package.

3. Unpack the archive and start the installation. The installation process will update the firmware of all Mellanox (NVIDIA) network cards found in the system.

   Installation of the minimum required set of packages:

   ./mlnxofedinstall --basic

   To update the firmware of network cards without driver installation:

   ./mlnxofedinstall --fw-update-only

   Reboot the system when finished.

   Important: add the --skip-distro-check option to mlnxofedinstall if OS version does not exactly match the version of MLNX_OFED package.

MITIGATOR setup

The specifics of working with the Mellanox (NVIDIA) driver is that they remain under Linux control, that is, the network interfaces remain in the system. However, you cannot transfer traffic through them when MITIGATOR is running.

1. Check that the output of the dpdk-devbind --status-dev=net command contains the required Mellanox (NVIDIA) network devices and that they are bound to the native mlx5_core driver.

2. If devices are present, but not bound to the mlx5_core driver, make sure that there is no attempt to bind these devices to another driver (for example, the PCI addresses of devices are NOT found in nics.conf).

Network card setup

1. Install the mstflint package, required for the setup scripts:

   apt install -y mstflint

2. Perform setup of Mellanox (NVIDIA) network cards firmware with the script mlx_firmware_tune. The script must be run once when installing a new network card. Requires a system reboot to apply changes.

   The script writes performance optimization settings into the card firmware and switches the mode of network ports from InfiniBand to Ethernet if supported.

3. Apply Mellanox (NVIDIA) network card settings for performance optimization with the script mlx_runtime_tune. Settings are active during the current session. The script must be run at the system startup by registering it in autoload.

4. Enable IOMMU pass-through mode for increased performance (Linux kernel parameter):

   iommu=pt