Backup

The following system components can be backed up:

cluster data - settings made through the web interface and API;
instance data - instance-specific settings;
metrics (graphs);
logs data.

Backup is possible only to the same version of MITIGATOR with which the backup was made.

Info

All configuration files in /srv/mitigator that are added or changed must be attached to the backup. This includes .env file, which is always configured during installation.

Backing up

A backup consists of a copy of the cluster data (backup.sql or backup_lite.sql) and a copy of the data of each instance (one or more backup_ownid.tgz).

Depending on your needs, a cluster data backup can be full (backup.sql) or lightweight (backup_lite.sql). A lightweight backup does not contain log or incident data.

Cluster Data Backup

Runs on the server hosting the database (primary if replication is configured).

Option A. With the MITIGATOR running:

docker-compose exec postgres sh -c 'backup > /tmp/backup.sql' && \
docker cp $(docker-compose ps -q postgres):/tmp/backup.sql .

docker-compose exec postgres sh -c 'backup_lite > /tmp/backup_lite.sql' && \
docker cp $(docker-compose ps -q postgres):/tmp/backup_lite.sql .

Option B. With the MITIGATOR stopped:

docker-compose up -d postgres && \
docker-compose exec postgres sh -c 'backup > /tmp/backup.sql' && \
docker cp $(docker-compose ps -q postgres):/tmp/backup.sql . && \
docker-compose down postgres

docker-compose up -d postgres && \
docker-compose exec postgres sh -c 'backup_lite > /tmp/backup_lite.sql' && \
docker cp $(docker-compose ps -q postgres):/tmp/backup_lite.sql . && \
docker-compose down postgres

After executing any of the above commands, you should manually verify that the backup was created without errors:

tail backup.sql

tail backup_lite.sql

Instance Data Backup

Executed on each of the instances:

tar -czf backup_ownid.tgz -C /var/lib/docker/volumes/mitigator_own_id/ .

Metrics Backup

Runs on the server that stores the metrics.

Info

The instruction is relevant for version v22.02 and higher. For older versions, please refer to the appropriate version instructions.

If MITIGATOR is stopped before starting the backup, you need to start the metrics storage module (docker-compose up -d clickhouse) and stop it after the backup procedure (docker-compose stop clickhouse).

docker-compose exec clickhouse sh -c 'backup' && \
docker cp $(docker-compose ps -q clickhouse):/tmp/metrics.tar.zst . && \
docker-compose exec -u root clickhouse rm /tmp/metrics.tar.zst

Restoring from a backup

The commands are given based on the fact that the files with the backup are located in the working directory /srv/mitigator, however, the location of the files can be anything:

on a remote server;
on USB storage devices;
in any system directory.

In this case, the commands must contain the full path to the backup files.

Cluster data recovery

Executed on the server that stores the database (primary if replication is configured).

Stop MITIGATOR:
```
docker-compose down
```

Warning

Removing the postgres volume will delete all MITIGATOR system settings, protection policies, logs, and users.
Make sure you have an up-to-date backup of your data before proceeding.

Delete an existing database:
```
docker volume rm mitigator_postgres
```

Run Postgres:

docker-compose up -d postgres && docker-compose logs -f postgres

Wait till the database system is ready to accept connections message, press Ctrl+C.
Transfer the backup to the container and restore the data:

docker cp backup.sql $(docker-compose ps -q postgres):/tmp && \
docker-compose exec postgres sh -c 'psql mitigator </tmp/backup.sql >/tmp/restore.log'

docker cp backup_lite.sql $(docker-compose ps -q postgres):/tmp && \
docker-compose exec postgres sh -c 'psql mitigator </tmp/backup_lite.sql >/tmp/restore.log'

Restoring Instance Data

Runs on every instance.

Deploy volume own_id:

tar -xzf backup_ownid.tgz -C /var/lib/docker/volumes/mitigator_own_id/ .

Run MITIGATOR:
```
docker-compose up -d
```

If errors appear during the recovery phase, technical support will need their text and the restore.log file, which can be taken from the container as follows:

docker cp $(docker-compose ps -q postgres):/tmp/restore.log .

Restoring Metrics