Settings

Default configuration can be changed via environment variables in /srv/collector/.env file (learn more).

Flow Collection

To turn on/off receiving IPv6 packets use:

  • COLLECTOR_ENABLE_IPV6 (default: true)

Flow packets are received on ports defined by the following environment variables:

  • COLLECTOR_NETFLOW_V5_PORT (default: 9555): the port used to receive NetFlow v5 protocol packets.
  • COLLECTOR_NETFLOW_V9_PORT (defalt: 9995): the port used to receive NetFlow v9 protocol packets.
  • COLLECTOR_IPFIX_UDP_PORT (default: 4739): the port used to receive IPFIX protocol packets over UDP.
  • COLLECTOR_IPFIX_TCP_PORT (default: 4739): the port used to receive IPFIX protocol packets over TCP.
  • COLLECTOR_SFLOW_PORT (default: 6343): the port used to receive sFlow v5 protocol packets.

ClickHouse

Collector saves flow from incoming packets to ClickHouse located using:

  • COLLECTOR_CLICKHOUSE_ADDRESS (default: clickhouse.collector:9000): address and port of the ClickHouse server.

By default, ClickHouse is configured to work on the server minimum configuration.
In order for ClickHouse to efficiently and safely use the entire available amount of RAM, you need to configure its limits.

  • COLLECTOR_CLICKHOUSE_RAM_RATIO_MAX (default: 0.7): the ratio to physical RAM for ClickHouse server amount of RAM used.
    Recommended value: 0.7.
    If ClickHouse on a single server - 0.9.

For example, 32 Gb of RAM is installed on the machine, the RAM limit for the ClickHouse server is 20.8 Gb:

COLLECTOR_CLICKHOUSE_RAM_RATIO_MAX=0.65
  • COLLECTOR_CLICKHOUSE_QUERY_RAM_MAX (default: 10Gi): the maximum amount of RAM to use for running a query on a single server. The recommended initial value is calculated by the formula: 0,9 × COLLECTOR_CLICKHOUSE_RAM_RATIO_MAX × (installed RAM).

This option prevents the exhaustion of available memory during parallel processing of queries.
When the number of incoming flows or the analyzed period increase, the value of COLLECTOR_CLICKHOUSE_QUERY_RAM_MAX should be increased.
If the number of parallel user sessions in MITIGATOR web UI or the number of API clients increases, the value of COLLECTOR_CLICKHOUSE_QUERY_RAM_MAX should be decreased.

For example, 32 Gb of RAM is installed on the machine, the RAM limit for the ClickHouse server is 22.4 Gb, the RAM limit for one query is 1 Gb:

COLLECTOR_CLICKHOUSE_RAM_RATIO_MAX=0.7
COLLECTOR_CLICKHOUSE_QUERY_RAM_MAX=1Gi

In this configuration, 12 DB queries can be processed in parallel. For the internal functions of the ClickHouse server (inserting data, background data processing), at least 10.4 Gb remains.

An extended analyzing period at the “Flow Analysis” page of MITIGATOR web UI requires more RAM to process flow records. If any kind of statistics disappears as soon as the analyzed period had been extended, you have to extend RAM size for ClickHouse or enable a file system usage mode (learn more).

Set the following environment variables to change the corresponding ClickHouse configuration options:

  • COLLECTOR_CLICKHOUSE_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY (default: 0)
  • COLLECTOR_CLICKHOUSE_MAX_BYTES_BEFORE_EXTERNAL_SORT (default: 0)

0 - dumping of temporary data is disabled.
Recommended value is calculated by the formula: COLLECTOR_CLICKHOUSE_QUERY_RAM_MAX / 2.

For example:

COLLECTOR_CLICKHOUSE_QUERY_RAM_MAX=10Gi
COLLECTOR_CLICKHOUSE_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY=5Gi

MITIGATOR Integration

Grafana

Grafana is used to display the metrics gathered by Collector. Grafana runs on port 3000, which can be accessed via a browser.

Default login/password: admin/admin.

At the first launch, select Data Sources, then select Collector.

Container restart policy

Container restart policy is controlled by the following environment variable:

  • CONTAINER_RESTART_POLICY (default: always)

Available values are listed in Docker documentation.