Reducing false-positives

False positives are created whenever Airlock Anomaly Shield flags legitimate sessions as anomalous. The causes of false positives can be very different – thus we have to choose the correct measures to successfully reduce the number of false positives.

  • Keep training data clean:

    • Training data should be based solely on legitimate traffic.

  • Create traffic exclusions to exclude non-legitimate traffic before gathering training data and ML model training.
    See also Optional configuration of Traffic Matchers.
  • Configure response rule exceptions and anomaly detection exclusions for Anomaly Shield applications:

    • Adding new response rule exceptions to Anomaly Shield applications can effectively reduce the number of blocks. However, creating exceptions will not influence the anomaly detection itself – logs will still be written whenever an anomaly has been detected.
    Anomaly detection exclusions allow bypassing certain traffic from being analyzed, which can be an appropriate measure to reduce false positives. Detection exclusions can also reduce the system load.

  • Configure response rule exceptions for Anomaly Shield applications if applicable.
    See also Section – Anomaly Detection.
  • Configure anomaly detection exclusions for Anomaly Shield applications if applicable.
    See also Traffic Matchers detail page.
  • Fine-tune the thresholds for anomaly detection to reduce the false positive rate:

    • Tuning anomaly detection thresholds greatly impact what behavior may result in what action. Keep in mind that the default threshold set is the result based on extensive research and is suitable for a wide variety of use cases.

  • Ensure the ML models have been trained with clean traffic data and that exceptions have been configured before testing/attempting any fine-tuning measures.
  • The default Anomaly Shield model thresholds can be tuned globally and for individual applications. See the table below.
  • Airlock Anomaly Shield evaluates sessions after the first 15 requests by default. This threshold can be increased so that the session evaluation happens at a later point of a session. See the table below.

Inspect the log messages

With clean training data in the ColdDB and the required enforcement exception rules configured, we have to inspect the WR-SG-NMLY-200 log messages in order to decide what to fine-tune:

Log-observation

Tuning measure

One ML model appears too aggressive and is active for most or all sessions.

This is a strong indicator for increasing the default model thresholds.

Airlock Anomaly Shield evaluates a session as anomalous within the first few requests, but the bit count decreases to non-anomalous values while the session is ongoing.

This is a strong indicator for increasing the number of requests within a session prior to the evaluation.

Threshold tuning

Anomaly indicator values are processed based on the set of thresholds in the Expert Settings during operation. For dry-run testing, new thresholds can be set in the analytics_indicator_thresholds.json file and be applied against ColdDB data. This allows you to simulate how changing the thresholds would affect the evaluation before applying new thresholds in operational mode.

  1. Get the current thresholds that are stored in the Expert Settings (Section – Security Gate).
    Default values:
    2. details...
    ################## 
# ANOMALY SHIELD # 
################## 
 
# DEFAULT INDICATOR THRESHOLDS 
# 
# (Context: global, AnomalyShield application) 
# 
# The indicator thresholds can be configured per model with 
# the following resources: 
AnomalyShield.Indicator.ConnectionMetrics.Threshold          "0.99" 
AnomalyShield.Indicator.GraphMetricsCluster.Threshold        "0.99" 
AnomalyShield.Indicator.IsolationForest.Threshold            "0.99" 
AnomalyShield.Indicator.MultipleCountries.Threshold          "0.99" 
AnomalyShield.Indicator.StatusCodeMeta.Threshold             "0.90" 
AnomalyShield.Indicator.TimingCluster.Threshold              "0.97" 
AnomalyShield.Indicator.QueryParameters.Threshold            "0.90"

    You can read out the threshold values of your current configuration with this command:

    /opt/airlock/ml-service/bin/airlock-ml-analytics export_thresholds

    Note that the displayed thresholds will be sorted in alphabetical order.

  2. Use your current thresholds in the /opt/airlock/ml-service/conf/analytics_indicator_thresholds.json file as a starting point for the analytics tool.
  3. Run the analytics tool in /opt/airlock/ml-service/bin/ against the ColdDB data and check which ML model is configured with an overly sensitive threshold.

    4. Example:
    /opt/airlock/ml-service/bin/airlock-ml-analytics --cold-db /var/airlock/ml-service/cold.db apply_models --query '{"bitcount": ">= 3"}' --output pattern bitcount

    Note that we filtered the output for "bitcount": ">= 3" in our example. Adjust the query as required.

  4. Gradually increase the thresholds that are found to be too sensitive.
    For example, you may change the IsolationForest.Threshold from 0.99 to 0.993 in the analytics_indicator_thresholds.json.
  5. Re-run the analytics tool against the ColdDB data and check if the change shows the desired effect on reducing false positives. If not, proceed the threshold tuning until the outcome of the analytics tool is acceptable.
  6. Apply the determined values either globally or as individual settings per application in the Expert Settings.

    7. Individual thresholds per application have the following Expert Settings syntax:

    AnomalyShield.Application.NAME-OF-AS-APPLICATION.Indicator.IsolationForest.Threshold            "0.993"
  7. After changing the Expert Settings, apply the new thresholds by activating the configuration.

Increasing the number of requests within a session

The minimal session length value can be increased if sessions are tagged anomalous after too few session requests and when the same session-bit count drops with additional requests. Note that the minimal required (request) aggregation value should be equally increased.
By doing so, session evaluation requires more requests within a session before the session is tagged.

  1. Go to:
    /opt/airlock/ml-service/devel/conf/ml.ini
  2. Increase the min_sess_length (default is 15) gradually, e.g., in +5 steps.
  3. Go to:
    /opt/airlock/ml-service/devel/conf/default_hyper_parameters.ini
  4. Increase the min_req_aggregation (default is 15) equally to min_sess_length.
  5. Re-train the models as described in the Training and model enforcement article and activate the configuration.
  6. Check if the changes show the desired effect on reducing false positives during operation. If not, further increase the values until the outcome is acceptable.