Backup

The following system components can be backed up:

  • cluster data - settings made through the web interface and API;
  • instance data - instance-specific settings;
  • metrics (graphs).

Backup is possible only to the same version of MITIGATOR with which the backup was made.

Backing up

A backup consists of a copy of the cluster data (backup.sql) and a copy of the data of each instance (one or more backup_ownid.tgz).

Cluster Data Backup

Runs on the server hosting the database (primary if replication is configured).

Option A. With the MITIGATOR running:

docker-compose exec postgres sh -c 'backup > /tmp/backup.sql'
docker cp mitigator_postgres_1:/tmp/backup.sql .

Option B. With the MITIGATOR stopped:

docker-compose up -d postgres
docker-compose exec postgres sh -c 'backup > /tmp/backup.sql'
docker cp mitigator_postgres_1:/tmp/backup.sql .
docker-compose down postgres

After executing any of the above commands, you should manually verify that the backup was created without errors:

tail backup.sql

Instance Data Backup

Executed on each of the instances:

tar -czf backup_ownid.tgz -C /var/lib/docker/volumes/mitigator_own_id/ .

Metrics Backup

Runs on the server that stores the metrics.

Info

The instruction is relevant for version v22.02 and higher. For older versions, please refer to the appropriate version instructions.

If MITIGATOR is stopped before starting the backup, you need to start the metrics storage module (docker-compose up -d clickhouse) and stop it after the backup procedure (docker-compose stop clickhouse).

docker-compose exec clickhouse sh -c 'backup'
docker cp mitigator_clickhouse_1:/tmp/metrics.tar.zst .
docker-compose exec clickhouse rm /tmp/metrics.tar.zst

Restoring from a backup

The commands are given based on the fact that the files with the backup are located in the working directory /srv/mitigator, however, the location of the files can be anything:

  • on a remote server;
  • on USB storage devices;
  • in any system directory.

In this case, the commands must contain the full path to the backup files.

Cluster data recovery

Executed on the server that stores the database (primary if replication is configured).

  1. Stop MITIGATOR:

    docker-compose down

  2. Delete an existing database:

    docker volume rm mitigator_postgres

  3. Run Postgres:

    docker-compose up -d postgres && docker-compose logs -f postgres

  4. Wait till the database system is ready to accept connections message, press Ctrl+C.

  5. Transfer the backup to the container and restore the data:

    docker cp backup.sql mitigator_postgres_1:/tmp
docker-compose exec postgres sh -c 'psql mitigator </tmp/backup.sql >/tmp/restore.log'

Restoring Instance Data

Runs on every instance.

  1. Deploy volume own_id:

    tar -xzf backup_ownid.tgz -C /var/lib/docker/volumes/mitigator_own_id/ .

  2. Run MITIGATOR:

    docker-compose up -d

If errors appear during the recovery phase, technical support will need their text and the restore.log file, which can be taken from the container as follows:

docker cp mitigator_postgres_1:/tmp/restore.log .

Restoring Metrics

Runs on the server that stores the metrics.

  1. Run Clickhouse (If it is stopped)

    docker-compose up -d clickhouse && docker-compose logs -f clickhouse | grep -qm1 "03-create-graphite_tagged-table.sql"

  2. Transfer backup to container and restore data

    docker cp metrics.tar.zst mitigator_clickhouse_1:/tmp
docker-compose exec clickhouse sh -c 'restore /tmp/metrics.tar.zst'

  3. Delete a backup archive from a container

    docker-compose exec clickhouse rm /tmp/metrics.tar.zst