Core isolation for performance optimization

By default, the CPU cores that work with network ports are also used by other subsystems. This can degrade performance and cause Input Errors pps/bps spikes on Port extX/intX graphs. You can take some of the load off these cores by preventing non-critical subsystems from running on them.

To do so:

In the core options, isolation of the packet processor cores is specified through the isolcpus=... and rcu_nocbs=... parameters. It is also recommended to add mitigations=off to disable core security patches.

These parameters are added to .env:

  • DPCPUS - list of cores allocated for packet processing (data-plane);
  • SRVCPUS - list of cores allocated for other subsystems (all other cores).

cpuset.yml file is created, applying the cpuset setting to all subsystems, and added to the list of settings in .env (COMPOSE_FILE=docker-compose.yml:docker-compose.override.yml:cpuset.yml):

version: "2.2"
services:
  data-plane:
    cpuset: ${DPCPUS}

  postgres:
    cpuset: ${SRVCPUS}

  backend:
    cpuset: ${SRVCPUS}

  nginx:
    cpuset: ${SRVCPUS}

  fwstats:
    cpuset: ${SRVCPUS}

  watchhog:
    cpuset: ${SRVCPUS}

  rts:
    cpuset: ${SRVCPUS}

  accesslog:
    cpuset: ${SRVCPUS}

  clickhouse:
    cpuset: ${SRVCPUS}

  graphite-clickhouse:
    cpuset: ${SRVCPUS}

  carbon-clickhouse:
    cpuset: ${SRVCPUS}

  carbonapi:
    cpuset: ${SRVCPUS}

  grafana:
    cpuset: ${SRVCPUS}

Example

  • Two CPUs of 10 physical cores with HyperThreading - a total of 40 logical cores:
CPU0: 0-9,20-29
CPU1: 10-19,30-39
  • All cores from each CPU are allocated for the data-plane, except for one, including paired ones:
0-8,10-18,20-28,30-38
  • The remaining cores are given to the rest of the subsystems:
9,19,29,39

grub:

GRUB_CMDLINE_LINUX_DEFAULT="... isolcpus=0-8,10-18,20-28,30-38 rcu_nocbs=0-8,10-18,20-28,30-38 mitigations=off"

.env:

DPCPUS=0-8,10-18,20-28,30-38
SRVCPUS=9,19,29,39

More about cpuset parameters