Using Fluentd

Fluentd (or td-agent) has many features and can collect and route different types of logs to different destinations. Filtering audit logs, user-trail, reporting records, different log levels, etc. is possible, for example. Airlock IAM can be integrated with Fluentd in both Container and traditional environments.

See the official Fluentd documentation for more details: https://docs.fluentd.org/installation

Below is an example installation on CentOS:

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh # You may also download this first and check it
systemd enable --now td-agent

It's possible and recommended to use a single instance of Fluentd for multiple instances/installations of Airlock IAM to simplify management.

When using the default Elasticsearch templates from Airlock, log messages are split into separate indexes automatically. This allows separate lifecycles to be implemented efficiently. Using Fluend it is possible to perform additional processing and routing upfront. Examples include routing audit logs to a different location for safer long-term storage.

Example splitting log records by "type":

/etc/td-agent/td-agent.conf.d/airlock-iam.conf

# Extract "type" from log record and use it as fluend tag
<match airlock-iam.**>
  @type rewrite_tag_filter
  <rule>
    key type
    pattern /(.*)/
    tag airlock-iam.$1
  </rule>
</match>

In td-agent.conf, you may include this file, and use match expressions, to refer to the tagged logs separately:

/etc/td-agent/td-agent.conf

@include /etc/td-agent/td-agent.conf.d/*

<match airlock-iam.**>
   type copy
   <store>
      @type elasticsearch
      host elasticsearch.example.com
      port 9200
      logstash_format true
      logstash_prefix airlock-iam-${tag_parts[1]}
      flush_interval 15
   </store>
</match>