Graphite on a separate server
Moving Graphite to a separate server allows you to split the load between two servers.
Until the metrics archive is transferred, only new graphs can be viewed in the MITIGATOR interface.
Further in the text Server1 is the server with MITIGATOR, Server2 is the server to which the transfer is being performed.
It is assumed that Server2 already has Docker and docker-compose installed and has a way to deliver the filebase from Server1 to Server2 (the amount of data can be more than 100 GB depending on the number of policies).
Versions of MITIGATOR on Server1 and on Server2 must match, for which:
Write a line with the version to the .env
file for Server1 and Server2:
VERSION=vXX.XX
Where vXX.XX
is a target version (i.e.: v20.08 or v20.12.1).
Preparing the storage server
-
Create
docker-compose.yml
on Server2:mkdir -p /opt/mitigator-graphite
Download
docker-compose.yml
:wget https://docs.mitigator.ru/v24.10/dist/grafbase/docker-compose.yml \ -O /opt/mitigator-graphite/docker-compose.yml
-
Create a service to run on Server2:
Download service file:
wget https://docs.mitigator.ru/v24.10/dist/docker-compose@.service \ -O /etc/systemd/system/docker-compose@.service
Activate service:
systemctl enable docker-compose@mitigator-graphite
-
Start services on Server2:
-
log in to docker.mitigator.ru
docker login docker.mitigator.ru
-
Go to work directory
cd /opt/mitigator-graphite
-
Download images
docker-compose pull
-
Run service
systemctl start docker-compose@mitigator-graphite
-
Switching graphs to the second server
-
On Server1, create
docker-compose.override.yml
or add the following to it:version: "2.2" services: backend: environment: BACKEND_GRAPHITE_URL: "http://192.0.2.10:3080/render/" fwstats: environment: FWSTATS_GRAPHITE_ADDRESS: "192.0.2.10:2003" clickhouse: scale: 0 graphite-clickhouse: scale: 0 carbon-clickhouse: scale: 0 carbonapi: scale: 0 grafana: scale: 0
Where
192.0.2.10
is the notional address of Server2, which should be replaced with a valid address. -
Apply parameters:
Attention! The backend service and BGP speaker will be restarted.
docker-compose up -d --scale clickhouse=1
This command launches a temporary ClickHouse instance to transfer data from it to a separate server. If migration is not required, you can skip to the «Finish» section.
Data transfer
-
Back up metrics on the old server [according to instructions]](https://docs.mitigator.ru/v24.10/en/maintenance/backup/#backup-metrics).
-
Move
metrics.tgz
to the new server. -
On the new server, restore the backup copy of the metrics [according to instructions]](https://docs.mitigator.ru/v24.10/en/maintenance/backup/#restore-metrics).
In the first step, set
VOLUME=mitigator-graphite_clickhouse
.
Finish
After the successful completion of all processes, you need to make sure that the archived metrics data is available from the MITIGATOR interface.
After that, stop and delete the ClickHouse container on Server1:
docker-compose stop clickhouse
docker-compose rm clickhouse
To free up space on Server1, you can delete the ClickHouse volume if it is no longer needed:
docker volume rm mitigator_clickhouse
Updating Server2
Before updating, it is highly recommended to make a backup.
-
Stop MITIGATOR:
systemctl stop docker-compose@mitigator-graphite
-
Update
docker-compose.yml
in/srv/mitigator
:wget https://docs.mitigator.ru/v24.10/dist/grafbase/docker-compose.yml \ -O /opt/mitigator-graphite/docker-compose.yml
-
Update
.env
:- Set
VERSION=v21.04.0
(or newer minor).
- Set
Graph migration when upgrading to version 21.04
-
Download the migration file update2104.sql to
/srv/mitigator
. -
Run ClickHouse:
docker-compose up -d clickhouse
-
Perform the migration:
docker-compose exec -T clickhouse clickhouse-client -mn < update2104.sql
Messages are written only in case of errors. In this case, the text of the message must be reported to the developers.
-
Make sure that the
docker-compose.override.yml
file on Server1 has: port 3080 specifiedversion: "2.2" services: backend: environment: BACKEND_GRAPHITE_URL: "http://192.0.2.10:3080/render/"
Completing the update
Run MITIGATOR service:
systemctl start docker-compose@mitigator-graphite