Fluentd (or td-agent) has many features and can collect and route different types of logs to different destinations. Filtering audit logs, user-trail, reporting records, different log levels, etc. is possible, for example. Airlock IAM can be integrated with Fluentd in both Container and traditional environments.
Installing Fluentd
See the official Fluentd documentation for more details: https://docs.fluentd.org/installation
Below is an example installation on CentOS:
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh # You may also download this first and check it systemd enable --now td-agent
It's possible and recommended to use a single instance of Fluentd for multiple instances/installations of Airlock IAM to simplify management.
Fluentd and Elasticsearch
When using the default Elasticsearch templates from Airlock, log messages are split into separate indexes automatically. This allows separate lifecycles to be implemented efficiently. Using Fluend it is possible to perform additional processing and routing upfront. Examples include routing audit logs to a different location for safer long-term storage.
Example splitting log records by "type":
/etc/td-agent/td-agent.conf.d/airlock-iam.conf
# Extract "type" from log record and use it as fluend tag <match airlock-iam.**> @type rewrite_tag_filter <rule> key type pattern /(.*)/ tag airlock-iam.$1 </rule> </match>
In td-agent.conf, you may include this file, and use match expressions, to refer to the tagged logs separately:
/etc/td-agent/td-agent.conf
@include /etc/td-agent/td-agent.conf.d/* <match airlock-iam.**> type copy <store> @type elasticsearch host elasticsearch.example.com port 9200 logstash_format true logstash_prefix airlock-iam-${tag_parts[1]} flush_interval 15 </store> </match>