Operational data store mode

Collector supports an operational data store mode. In ODS mode, the collector accumulates data in a dedicated storage and provides this data for analysis. The ODS mode helps to speed up data access for end users, to reduce ClickHouse load and guarantees availability of aggregated data in case of ClickHouse outage. This mode is also could be used for minimal installations of the collector without ClickHouse installation.

Aggregated statistics

In ODS mode, the collector aggregates the next data:

• top source IP-addresses
• top destination IP-addresses
• top source ports
• top destination ports
• top packets length
• top TCP-flags

If a GeoIP database was set up, the next data also will be available:

• top source countries
• top destination countries
• top source autonomous systems
• top destination autonomous systems

Also, top source and destination IP-addresses will contain geo-data too.

Setup of operational data store mode

By default, the ODS mode is disabled.

Enabling of the mode is leading to additional RAM consumption. With recommended settings, hugepages extra usage is 80 MB initially and 1.5 MB per policy, and ordinary RAM extra usage is 120 MB per policy. In the ODS mode, top statistics are available only for General Protection and for up to 64 policies.

To enable the ODS mode, set up the next variables in /srv/collector/.env file:

COLLECTOR_STATS_TOP_SIZE=15
COLLECTOR_BACKEND_STATS_PERIOD="1h"

COLLECTOR_STATS_TOP_SIZE sets the maximum number of top elements stored. Recommended value: 15.

COLLECTOR_BACKEND_STATS_PERIOD sets the maximum duration of statistics storing. Recommended value: 1h