Administrator Tasks

  1. Ensure network security of the server.
  2. Monitor free disk space.
  3. Manage logs.
  4. Periodically backup database.

It is recommended to set up monitoring of the status of the server on which MITIGATOR is installed using Zabbix, Nagios or a similar solution.

Network security

In the basic package, the system is designed for the management interface to be located in a secure network segment and does not in any way restrict access for unprotected additional interfaces, such as Grafana.

If the management interface is in a public segment, it is necessary to restrict access to all forwarded ports by system means (for example,, iptables), using a firewall or other solutions.

Using the capabilities of the Linux network subsystem, you can configure routing independently for each management interface and address. It is recommended to use separate addresses for the management interface (SSH, API, web), for BGP, for polling servers in the GAME and SOUR countermeasures. To do this, you either need to own two or more IPs, or some of the networks must be internal: for example, the provider’s BGP interface can be accessed at an internal address. Even if there is only one physical interface, multiple addresses can be assigned to it. Example for /etc/network/interfaces, where 192.0.2.2 is a main public address and 172.16.0.2 is located in the provider network 172.16.0.0/24:

auto eth0
iface eth0 inet static
    # main address, usually for SSH, API and web
    address 192.0.2.2/24
    gateway 192.0.2.1

    # block for BGP link address
    up ip address add 172.16.0.2/24 dev eth0
    down ip address delete 172.16.0.2/24 dev eth0

Disk space

For the normal functioning of the system over a significant period of time (a year or more), up to 100 GB or more of free disk space is required.

You can find out the space occupied by docker elements (images, containers, volumes) using the following command:

docker system df

During operation, if you do not delete old and unused elements, free disk space will be gradually consumed (primarily by old images).

You can clear the space occupied by currently unused elements with the following command (needs confirmation):

docker system prune

In addition, the space is occupied by Postgres (system settings) and Graphite (graphics) databases.

This space does not decrease during the operation of the MITIGATOR, and the amount of space occupied will only increase.

You can find out the space occupied by the databases with the following command:

du -hs \
    /var/lib/docker/volumes/mitigator_postgres \
    /var/lib/docker/volumes/mitigator_clickhouse

To clear Clickhouse logs:

docker-compose exec clickhouse bash
clickhouse-client -q "SELECT name FROM system.tables WHERE name LIKE '%log%';" | xargs -I{} clickhouse-client -q "TRUNCATE TABLE system.{};"

Logs

There are two aspects to log management:

  • logs take up disk space;
  • logs may be needed by developers to solve issues.

Logging setup

Through dockerd parameters, you can limit the size of files, set up rotation, and in more complex cases, send logs to ELK, etc. Basic recommended option: daemon.json.

When changing daemon.json, you need to restart the Docker daemon and MITIGATOR:

systemctl restart docker
systemctl restart mitigator

Saving logs

The logs are tied to the container, so when it is recreated, they are destroyed. Commands leading to re-creation of containers:

  • docker-compose down, as the containers are deleted.
  • docker-compose up, if the images have been updated.
  • systemctl restart mitigator, as it executes the commands above.

The docker-compose restart <service> command does not lose logs.

If you need to restart and still save logs for debugging, you can do this as follows:

cd /var/lib/docker/containers
find . -name '*-json.log' \
    -exec tar czf /srv/mitigator/logs_$(date +%y%m%d).tgz {} +
cd -

Backup

System administrators are required to have an up-to-date backup of the MITIGATOR database.

It is mandatory to backup database before each update procedure. It is recommended to organize a periodic backup process.

It is a good practice to backup the configuration directory /srv/mitigator before every configuration update.