Troubleshooting

Drivers not loading (modprobe)

Symptoms:

  • When loading a module:

    modprobe: FATAL: Module igb_uio not found in directory /lib/modules/4.9.0-6-amd64

  • When installing a package:

    Module build for kernel 4.9.0-6-amd64 was skipped since the
kernel headers for this kernel does not seem to be installed.

You need to install the linux-headers-amd64 (Debian) package and make sure that the kernel version corresponding to the version of this package is loaded, then run:

dkms autoinstall

Docker and docker-compose not working

By default, these commands only work for users in the docker group and for root. This is a standard Docker security measure. Solution:

usermod -aG docker $USER && \
newgrp docker

MITIGATOR does not start (unknown log opt)

When starting MITIGATOR, an error like the following appears:

Error response from daemon: unknown log opt 'max-file' for syslog log driver

The base docker-compose.yml configuration sets log rotation options that are not compatible with logging drivers other than the default json-file (for example, syslog or journald). For the fix, see Logging setup.

Packet processor not available

Such a message appears in the web interface if the package processor did not start or stopped. Diagnostics (from /srv/mitigator):

docker-compose logs dataplane

Below are some typical problems and typical messages.

EAL: Cannot get hugepage information

Message:

EAL: No free hugepages reported in hugepages-2048kB
EAL: No free hugepages reported in hugepages-1048576kB
EAL: FATAL: Cannot get hugepage information.
EAL: Cannot get hugepage information.

Reason: Hugepages not allocated or fully occupied.

Check for available hugepages and make sure HugePages_Total and Hugepagesize are the same as configured, and HugePages_Free is equal to HugePages_Total (or differs by no more than a few pages):

grep Huge /proc/meminfo

EAL: set_mempolicy failed: Invalid argument

Message:

EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 1
EAL: set_mempolicy failed: Invalid argument
EAL: alloc_seg(): allocation happened on wrong socket (wanted 1, got 0)
EAL: Ask a virtual area of 0x40000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x40000000)
EAL: attempted to allocate 1 segments, but only 0 were allocated

Reason: Hugepages were not allocated on all NUMA nodes used by the application on a multiprocessor platform.

Check command output (for a platform with two NUMA nodes):

ls /sys/devices/system/node/node0
ls /sys/devices/system/node/node1

The output should contain the hugepages and memory directories and should not differ much. If there are no such directories, check the machine for the physical presence of memory on both CPUs. View memory information in BIOS, check NUMA settings in BIOS, check memory and NUMA settings in OS.

no port found

Message:

/dataplane/etc/dataplane.conf:0 no port found with name '04:00.0'

or

/dataplane/etc/dataplane.conf:0 no port found with pci address '04:00.0'

You need to make sure that the device address 04:00.0 is correct, and that the device is bound to the DPDK driver in the same way as during system preparation.

If the device is bound, look for other errors in the log, for example:

EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL:   0000:04:00.0 VFIO group is not viable! Not all devices in IOMMU
group bound to VFIO or unbound EAL: Requested device 0000:04:00.0 cannot be used

This error means that not all devices in the same IOMMU group are bound to the vfio-pci driver. To work through vfio-pci, you need to bind all ports of one card to vfio-pci, or switch to another driver.

Crash

When looking at the docker-compose logs dataplane logs, the message is:

Segmentation fault (core dumped)

To resume work, run the command:

docker-compose up -d dataplane

Further, it is recommended to contact the developers for the purpose of accurate diagnosis and correction and send them:

  1. Description of the situation in which the issue occurred (actions with MITIGATOR, incoming traffic).

  2. Description of the server hardware configuration:

    • CPU and network cards model;
    • memory size and hugepages configuration;
    • Linux kernel version, network card drivers and their versions.

  3. The /srv/mitigator/dataplane.conf file.

  4. The dataplane.stacktrace file, which is stored in /var/lib/docker/volumes/mitigator_coredumps/_data/dataplane.stacktrace by default.

  5. Core dump (a snapshot of process memory), which is stored by default in /var/lib/docker/volumes/mitigator_coredumps/_data/core. Its size is gigabytes, but it usually compresses well.

    On some distributions (eg Ubuntu) the core file does not appear because the crash handling is configured differently. You can check the setting like this:

    cat /proc/sys/kernel/core_pattern

    The default on Linux is core. If it’s different:

    1. Remember original value:

      cat /proc/sys/kernel/core_pattern > /tmp/old_core_pattern

    2. Replace it with core:

      echo core | tee /proc/sys/kernel/core_pattern

    3. Reproduce the issue, get the core file from the path above.

    4. Restore original value:

      cat /tmp/old_core_pattern | tee /proc/sys/kernel/core_pattern

Related Content