Increasing Airlock Anomaly Shield evaluation throughput with additional worker processes

Prerequisites

Before enabling Airlock Anomaly Shield, make sure that you have enough free CPU resources to handle even peak loads of the Airlock Gateway. A good indicator is when the CPU load remains below 50%.

Mind the Performance considerations on load and configuration before increasing the number of worker processes.

Introduction

The Airlock Anomaly Shield evaluation performance is limited by the number of parallel machine learning service worker processes. By default, the number of worker processes is 2, but can be increased if your system configuration allows assigning more exclusive machine learning service processes. Note that assigning additional worker processes will occupy CPU cores.

  1. We recommend the following steps:
  2. Check the system performance with Airlock Anomaly Shield enabled. Calculate a metric that examines how many evaluations take place per session per minute.
  3. Increase the number of worker processes in the /opt/airlock/ml-service/conf/ml.ini file accordingly.

Step 1 – Calculate the Anomaly Shield evaluation performance

  1. Configure log session anomaly details
  2. Go to:
    Application Firewall >> Anomaly Shield Applications >> Anomaly Detection and Response
  3. Set Log session anomaly details to When raw session anomaly values change.
    4. Anomaly Shield raw session values change
  1. Calculate the Anomaly Shield evaluation performance
  2. Go to:
    Log & Report >> Log Viewer
  3. Compare the amount of WR-SG-SUMMARY logs with WR-SG-NMLY-200 per session for an interval of e.g. 30 minutes.

    4. The ratio of the two log messages shows how many requests happen per evaluation of a session.

  4. Calculate how many Anomaly Shield evaluations take place per session and minute.

Step 2 – Increase the number of worker processes

  1. Edit the number of worker processes, according to the evaluation performance, e.g. to processes = 3:
    /opt/airlock/ml-service/conf/ml.ini
    2. ... 
# Set the number of processes to fork 
processes = 3 
...
  2. Restart the service:
    3. systemctl restart airlock-ml-service.service

Re-calculate the Airlock Anomaly Shield evaluation performance (see Step 1 – Calculate the Anomaly Shield evaluation performance). Note that the average CPU load should remain below 50%. Increase the number of worker processes if required and if the CPU load allows.