Core isolation for performance optimization

By default, the CPU cores that work with network ports are also used by other subsystems. This can degrade performance and cause Input Errors pps/bps spikes on Port extX/intX graphs. You can take some of the load off these cores by preventing non-critical subsystems from running on them.

To do so:

  1. Specify isolation of the packet processor cores in the core options through the isolcpus=... and rcu_nocbs=... parameters. It is also recommended to add mitigations=off to disable core security patches.

  2. Add these parameters to .env:

    • DATA_PLANE_CPUS — list of cores allocated for packet processing (data-plane);
    • CONTROL_PLANE_CPUS — list of cores allocated for other subsystems (all other cores).

  3. Download docker-compose.cpuisol.yml, that applies cpuset option to all subsystems:

    wget https://docs.mitigator.ru/v23.08/dist/docker-compose.cpuisol.yml -O /srv/mitigator/docker-compose.cpuisol.yml

  4. Add it to the COMPOSE_FILE list in .env:

    COMPOSE_FILE=docker-compose.yml:docker-compose.cpuisol.yml

Example

  • Server with two CPUs of 10 physical cores with HyperThreading — a total of 40 logical cores:
CPU0: 0-9,20-29
CPU1: 10-19,30-39
  • All cores, except for one, from each CPU are allocated for the data-plane, including HT-pairs:
0-8,10-18,20-28,30-38
  • The remaining cores are given to the rest of the subsystems:
9,19,29,39

grub settings:

GRUB_CMDLINE_LINUX_DEFAULT="... isolcpus=0-8,10-18,20-28,30-38 rcu_nocbs=0-8,10-18,20-28,30-38 mitigations=off"

.env settings:

DATA_PLANE_CPUS=0-8,10-18,20-28,30-38
CONTROL_PLANE_CPUS=9,19,29,39

More about cpuset parameter