Customize Elasticsearch data archiving

This article describes how logs are cleaned and how some parameters may be customized in Airlock Gateway.

How log cleanup works

Airlock Gateway stores its log files in Elasticsearch. A fresh index is created every day, storing all logs of the corresponding day. As soon as the usage in /var exceeds a specific trigger limit or when the log files exceed a certain age in days, the corresponding indices are deleted to a certain percentage of disk space to ensure enough capacity is available in /var.

Default settings

Description

PERCENT_TO_TRIGGER=75

If disk usage exceeds the configured percentage, a cleanup will be triggered.

PERCENT_TO_KEEP=70

Cleanup deletes the oldest indices until the disk usage is at the configured percentage.

MAX_DAYS_TO_KEEP=365

Regardless of the trigger levels, all indices older than the configured number of days will be deleted.

Custom configuration

The default settings suit most system configurations. However, for a high-traffic system with low disk drive capacity, there might not be enough space left to store full logs of the current day. Some companies also have a policy that restricts the storage of logs records to a shorter period.

  1. In both cases, you need to adjust the settings to the required values as follows:
  2. Create an airlock-elasticsearch-space-nurse.cfg config file:
    3. copy
    vi /opt/airlock/custom-settings/monitor/airlock-elasticsearch-space-nurse.cfg
  3. Add the default settings to the file:
    4. copy
    PERCENT_TO_TRIGGER=75 
PERCENT_TO_KEEP=70 
MAX_DAYS_TO_KEEP=365
  4. Adjust the settings to your needs.
  5. Restart the airlock-elasticsearch-space-nurse service.
    6. copy
    systemctl restart airlock-elasticsearch-space-nurse