Performance tuning and scaling best practices

There are various aspects that influence the performance and ability to scale an Airlock IAM system depending on the precise usage scenario. This article highlights the most common bottlenecks.

In Airlock IAM audit log signing is disabled by default. The signature ensures that manipulations of log messages are detectable. The generation of log sequence numbers and digital signatures limits scaling with the number of CPU cores. Disabling audit log signing can speed up the whole system significantly. Audit log messages are still written - but not signed anymore.

0. To change audit log signing:
1. Open the instance configuration (e.g. instances/auth/instance.properties) in an editor.
2. To disable set the property "iam.audit-log.signing.enabled" to false.
3. To enable set the property "iam.audit-log.signing.enabled" to true.
4. Restart the IAM instance to activate the changes.

The Data sources (databases, directories) section lists the supported databases.

• Choose your database according to your requirements:
• Some of them are known to scale well with the number of users or tokens.
• Others, like the H2 database, are fine for smaller scenarios only.

The supported H2 database has limitations, therefore we recommend using it only for testing purposes and for small production scenarios with some hundreds of users.

• Limitations of the H2 database:
• H2 uses only one index per logical table at a time in each query.
• The H2 cluster feature imposes additional limitations on performance (result sets are completely loaded into memory).

The chosen database, as well as the database host, have to be monitored to detect unexpected increasing system load, a high proportion of I/O wait, or inefficient queries due to missing indexes.

To scale well, a database needs indices that are based on the most frequent use cases. Since Airlock IAM is highly configurable regarding database fields, the indices have to be set with the knowledge of the field configuration and usage.

IAM defines indices and primary keys in the database schema setup scripts. The database schemas are not updated automatically when updating the Airlock IAM software. Changes to the database schemas have to be performed manually.

The schema files and the schema update files per database can be found here:
Data sources (databases, directories)

DB systems use optimizers to tune queries on their own. The decisions of the query optimizer are base on database usage statistics. The statistics have to be up-to-date to result in the best decisions.

We strongly recommend configuring the DB system to create new statistics regularly.

Connections from Airlock IAM to the DB are handled by connection pools.

• The idea is to run only as many parallel requests in the database as it can handle at a time:
• Using too few DB connections will result in a bottleneck (wait events in IAM).
• Using too many DB connections can result in DB wait events and lower the overall performance as well.

Each configured database connection has its own pool size. A separate connection pool is used by each module (Loginapp, Adminapp, SOAP Interface, Service Container, Transaction Approval) and connection pool configuration (e.g. pool size). The Loginapp typically needs more database connections than the Adminapp.

As a rule of thumb, the developers of the "Hikari" connection pool recommend limiting the number of connections to twice the number of CPU cores of the database server plus the number of disk spindles.

See https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing for more information about connection pool sizing.

Database vendors provide JDBC drivers for specific versions of their databases. JDBC drivers are not bundled with Airlock IAM and must be downloaded and deployed separately. It is essential to use the latest JDBC driver that matches the database version currently in use. Unexpected behavior including degraded performance can result from an inappropriate driver version.

The choice of persistency plugins has an influence on the performance:

• Whenever possible favor using User Store plugins over User Persister plugins. This is especially true for the user management in the Adminapp.
• If combining multiple persistence systems (e.g. the IAM database and an external system via a custom plugin), minimize calls to potentially slow external systems. Typically, not all calls in all plugins need to involve external persistence systems.

Some hash algorithms, such as scrypt are expensive by design in order to make brute-force attacks on hash databases unfeasible, i.e. increase the security of the hashed data. On the other hand, choosing an expensive hash algorithm limits the number of requests that a system is able to process per time unit. Other hash algorithms, such as sha256 are pretty fast and therefore easier to brute-force.

Airlock IAM uses hashes for various use cases. Depending on the use case a different hash algorithm is suitable.

• We recommend:
• Scrypt for passwords, secret questions, and IAK letters.
• SHA-256 for OAuth, Open ID Connect, and matrix cards.

See also Selection and parametrization of hash functions for further information and tuning options.

New Airlock IAM versions are often bringing additional performance optimizations. We recommend checking for newer versions and updating/upgrading on a regular basis.

For Docker deployments, this information does not apply. Please refer to the Docker documentation or the documentation of your orchestration software (Kubernetes, OpenShift, etc.).

Under rare circumstances, such as during SAML performance and integration testing, a large peak number of requests can generate exception errors (Too many open files) in the IAM log files. In this case, the currently configured limits for file descriptors have been exceeded and can be adjusted for the corresponding service.

Airlock IAM uses systemd to start and stop IAM and IAM services, see also Starting and stopping Airlock IAM (system service integration). Because of this, file descriptor limits for services have to be configured for systemd and not by changing ulimit values.

The IAM CLI allows adjusting the file descriptor limit per service using the option -l or --limit-file-descriptors as in the following example:

copy

iam systemd -i auth -l 8192

This creates a systemd service file with a file descriptor limit of 8192 for the auth service.

• Assumed Hardware:
• Server with 2 CPUs each with 8 cores at 2.5 GHz.

• Assumed Configuration:
• Passwords with scrypt hashes as the first factor.
• mTAN as the second factor.
• The audit log signing is switched off.
• The database connection pool has a size of 16 and fits well the performance of the DB server.
• The suggested indices are created.

• Expected Performance:
• 50-90 entire login/logout roundtrips per second

Airlock IAM scales well with the number of CPUs and cores. However, CPU threads won't help, since the system is CPU-bound (scrypt). CPU threads are beneficial if a CPU core has to wait for I/O.

For further scaling, it is possible to load-balance multiple Airlock IAM behind Airlock Gateway.

• Data sources (databases, directories)
• Selection and parametrization of hash functions
• https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing