Graphite on a Separate Server

Moving Graphite to a separate server allows you to split the load between two servers.

Until the metrics archive is transferred, only new graphs can be viewed in the MITIGATOR interface.

Further in the text Server1 is the server with MITIGATOR, Server2 is the server to which the transfer is being performed.

It is assumed that Server2 already has Docker and docker-compose installed and has a way to deliver the filebase from Server1 to Server2 (the amount of data can be more than 100 GB depending on the number of policies).

Versions of MITIGATOR on Server1 and on Server2 must match, for which:

Write a line with the version to the .env file for Server1 and Server2:

VERSION=vXX.XX.X

Where vXX.XX.X is a target version (i.e.: v20.12.1).

Preparing the storage server

  1. Create docker-compose.yml on Server2:

    mkdir -p /opt/mitigator-graphite

    Download docker-compose.yml:

    wget https://docs.mitigator.ru/master/dist/grafbase/docker-compose.yml \
    -O /opt/mitigator-graphite/docker-compose.yml

  2. Create a service to run on Server2:

    Download service file:

    wget https://docs.mitigator.ru/master/dist/docker-compose@.service \
    -O /etc/systemd/system/docker-compose@.service

    Activate service:

    systemctl enable docker-compose@mitigator-graphite

  3. Start services on Server2:

    • log in to docker.mitigator.ru

      docker login docker.mitigator.ru

    • Go to work directory

      cd /opt/mitigator-graphite

    • Download images

      docker-compose pull

    • Run service

      systemctl start docker-compose@mitigator-graphite

Switching graphs to the second server

  1. On Server1, create docker-compose.override.yml or add the following to it:

    services:
  backend:
    environment:
      BACKEND_GRAPHITE_URL: "http://192.0.2.10:3080/render/"

  fwstats:
    environment:
      FWSTATS_GRAPHITE_ADDRESS: "192.0.2.10:2003"

  clickhouse:
    scale: 0

  graphite-clickhouse:
    scale: 0

  carbon-clickhouse:
    scale: 0

  carbonapi:
    scale: 0

  grafana:
    scale: 0

    Where 192.0.2.10 is the notional address of Server2, which should be replaced with a valid address.

  2. Apply parameters:

    Attention! The backend service and BGP speaker will be restarted.

    docker-compose up -d --scale clickhouse=1

    This command launches a temporary ClickHouse instance to transfer data from it to a separate server. If migration is not required, you can skip to the «Finish» section.

Data transfer

  1. Back up metrics on the old server according to instructions.

  2. Move metrics.tgz to the new server.

  3. On the new server, restore the backup copy of the metrics according to instructions.

    In the first step, set VOLUME=mitigator-graphite_clickhouse.

Finish

After the successful completion of all processes, you need to make sure that the archived metrics data is available from the MITIGATOR interface.

After that, stop and delete the ClickHouse container on Server1:

docker-compose stop clickhouse && \
docker-compose rm clickhouse

To free up space on Server1, you can delete the ClickHouse volume if it is no longer needed:

docker volume rm mitigator_clickhouse

Updating Server2

Info

Before updating, it is highly recommended to make a backup.

  1. Stop MITIGATOR:

    systemctl stop docker-compose@mitigator-graphite

  2. Update docker-compose.yml in /srv/mitigator:

    wget https://docs.mitigator.ru/master/dist/grafbase/docker-compose.yml \
    -O /opt/mitigator-graphite/docker-compose.yml

  3. Update .env:

    • Set VERSION=v21.04.0 (or newer minor).

Graph migration when upgrading to version 21.04

  1. Download the migration file update2104.sql to /srv/mitigator.

  2. Run ClickHouse:

    docker-compose up -d clickhouse

  3. Perform the migration:

    docker-compose exec -T clickhouse clickhouse-client -mn < update2104.sql

    Messages are written only in case of errors. In this case, the text of the message must be reported to the developers.

  4. Make sure that the docker-compose.override.yml file on Server1 has: port 3080 specified

        services:
      backend:
        environment:
          BACKEND_GRAPHITE_URL: "http://192.0.2.10:3080/render/"

Completing the update

Run MITIGATOR service:

systemctl start docker-compose@mitigator-graphite

Related Content