Log analyzer

Analyzes web server logs in access.log format and calculates statistics. It allows to identify anomalies and attacking IP addresses. Can be used when writing filtering rules in LOGAN.

The mechanism accepts a file with Web server Access logs in “File for analysis” field and a log format in “NGINX log-format” field.

The “Output threshold” field specifies the minimum number of repetitions of the calculated parameter to be included in the report.

Report

The report contain sections:

• Total;

  • File — file name;
  • Number of records — number of records in the file;
  • Rps — average requests per second rate.

• Statistics for IPs;

  • Total unique IPs — number of unique IP addresses;
  • Total count — number of records for each IP address;
  • IPs by rps — rate of requests per second received from each IP address;
  • Total paths without params count — total number of paths without parameters;
  • IPs by paths without params count — number of paths without parameters for each IP address;
  • Total paths with params count — total number of paths with parameters;
  • IPs by paths with params count — number of paths with parameters for each IP address;
  • Total User-Agents count — number of unique User-Agents;
  • IPs by User-agents count — User-Agent distribution by IP addresses;
  • Total referers count — number of unique referers;
  • IPs by referers count — referrer distribution by IP addresses;
  • Total request time — total time to process requests;
  • IPs by total request time — distribution of request processing time by IP addresses;
  • Total bytes sent — total number of bytes sent;
  • IPs by total bytes sent — distribution of sent bytes by IP addresses;
  • IPs by statuses — distribution of status codes by IP addresses;
  • IPs by methods — distribution of methods by IP addresses.

• Statistics for statuses;

  • Total unique statuses — number of unique status codes;
  • Total count — total number of logs with status code;
  • Statuses by count — number of logs for each status code;
  • Statuses by rps — average rate of requests with a certain status code.

• Statistics for paths;

  • Total unique paths without params — total number of unique paths without parameters;
  • Total count — total number of logs with paths without parameters;
  • Paths without params by count — frequency of a unique path without parameters occurrence in requests;
  • Total IPs count — total number of IP addresses that sent a request with a path without parameters;
  • Paths without params by IP count — number and percentage of IP addresses that sent each request without parameters;
  • Total unique paths with params — total number of logs with paths in which parameters are specified;
  • Total count — total number of logs with paths in which parameters are specified;
  • Paths with params by count — frequency of a unique path with parameters occurrence in requests;
  • Total IPs count — total number of IP addresses that sent a request with a path in which the parameter is specified;
  • Paths with params by IP count — number and percentage of IP addresses that sent each request with parameters;

• Statistics for User-Agents;

  • Total unique User-Agents — number of unique User-Agents;
  • Total count — total number of requests with User-Agent specified;
  • User-Agents by count — number of requests with each User-Agent;
  • User-Agents by rps — average rate of requests received with a given User-Agent.

• Statistics for referers;

  • Total unique referers — number of unique referers;
  • Total count — total number of requests with referer specified;
  • Referers by count — number of requests with each referer;
  • Total IPs count — total number of IP addresses that sent a request with referer;
  • Referers by IP count — number and percentage of IP addresses that sent that sent each request with referrer;
  • Referers by rps — average rate of requests received with a given referer.

• Statistics for hostnames;

  • Total unique hosts — number of unique hostnames;
  • Total count — total number of requests with hostname;
  • Hostnames by count — number of requests with each hostname.

• Statistics for schemes;

  • Total unique schemes — number of unique schemes;
  • Total count — total number of requests with the scheme specified;
  • Schemes by count — number of requests for each scheme.