Network between instances

Principle of Operation

The following steps assume that an instance of MITIGATOR has already been installed. Otherwise, perform installation using one of the following methods.

Interaction between instances is arranged through Wireguard. A virtual network (VPN) is created between instances.

Each instance has a key pair: a private key and a public key. The private key is stored only on its instance in vpn-private.conf. The public keys of all instances are listed in vpn-public.conf, which must be the same on all instances. It also contains the addresses of the instances and their addresses in the VPN.

The VPN instance address is also specified as MITIGATOR_VPN_ADDRESS in .env.

VPN is maintained by gateway and gateway-dataplane services. Without clustering, they do not configure VPN. With clustering enabled, they configure VPN according to vpn-private.conf, vpn-public.conf, MITIGATOR_VPN_ADDRESS.

Info

If you need to change gateway settings other than *.conf, you need to completely restart the instance (docker-compose down && docker-compose up -d).

See below for a specific command to update the VPN configuration.

Mellanox NIC Case

When using Mellanox NICs, dataplane and gateway-dataplane work without network isolation. The VPN between packet processors will also not be isolated from the host. Take care to pick MITIGATOR_VPN_ADDRESS not residing in any existing network that needs to be reachable from the host.

When VPN is enabled, gateway-dataplane creates wg0 interface on the host. This interface name can be changed via DATA_PLANE_VPN_DEVICE in .env.

The VPN between packet processors uses port 4568 (UDP). Ensure all instances can access this port of each other. This port number can be changed via DATA_PLANE_VPN_PORT in .env.

System Preparation

Kernel Module

  • Debian 10 requires installation of a ported package:

    echo deb http://deb.debian.org/debian buster-backports main > /etc/apt/sources.list.d/buster-backports.list
apt update
apt install -y linux-headers-amd64 wireguard-dkms

  • Debian 11+ requires installation of a regular package:

    apt install -y linux-headers-amd64 wireguard-dkms

  • Ubuntu 20.04+ includes Wireguard in the base distribution, no additional steps required.

You can check support with the modprobe wireguard command. If nothing is printed in response, the module is available. In this case, it is enough to configure its automatic loading. Otherwise, a reboot is required.

To add wireguard to automatic download:

echo wireguard >> /etc/modules-load.d/mitigator.conf

Tool Installation

Install wg utility:

apt install -y wireguard-tools

Instance Setup

All of the files are created in /srv/mitigator catalog.

If more than the first instance is configured, the vpn-public.conf file must be taken from any of the configured instances to complete it.

  1. Create the private key (Resulting example: yDPg5doavYH7fdD86nt+cOzSBL4znVZcrcrJwjY/Xmw=):

    wg genkey

  2. Write the key in vpn-private.conf:

    [Interface]
ListenPort = 4567
PrivateKey = yDPg5doavYH7fdD86nt+cOzSBL4znVZcrcrJwjY/Xmw=

    The specified port 4567 must be open for UDP traffic.

  3. Get public key from private key (Resulting example: acfzxE6ZsiYE4jIqsBicOt7oT8ZuKhxBvuz0+6JxiEc=):

    echo 'yDPg5doavYH7fdD86nt+cOzSBL4znVZcrcrJwjY/Xmw=' | wg pubkey

  4. Add a section with the public key and instance addresses to vpn-public.conf (create a file if this is the first instance):

    [Peer]
PublicKey = acfzxE6ZsiYE4jIqsBicOt7oT8ZuKhxBvuz0+6JxiEc=
AllowedIPs = 10.8.3.1/32
Endpoint = 192.0.2.1:4567

    10.8.3.1 is an instance address inside VPN. Must be unique among all instances. All addresses must be within the same /24 network (default).

    192.0.2.1:4567 is the external address of the instance and the port configured above. Other instances will send UDP packets to this address and port.

  5. Add the address of the instance inside the VPN to .env:

    MITIGATOR_VPN_ADDRESS=10.8.3.1

    It must match the one configured in vpn-public.conf. Also, this address must be specified in the instance settings in the MITIGATOR web interface.

  6. Reboot MITIGATOR:

    docker-compose down && docker-compose up -d

Cluster Setup After Adding an Instance

After adding a new instance to the vpn-public.conf file, or when changing vpn-private.conf, you need to make changes on all instances.

On each instance, you need to update the VPN configuration without restarting:

docker-compose exec gateway reconfigure
docker-compose exec gateway-dataplane reconfigure