Kafka Message Broker Connector
This plugin uses the Apache Kafka Java client library to produce so-called records and send them to the configured broker(s). The configurable properties expose the main settings of the producer part of the library. The properties exposed here, as well as further configurable properties, can be found in the official Kafka Producer Configs documentation.
Events and Kafka Records The following description explains how the events produced by Airlock IAM are mapped to Kafka records. Each event sent to Kafka is published as a record in a topic. A record consists of three main parts:
- record key: Identifies the source of the event, configured by the 'Record Key' property.
- record value: Contains the actual event data in JSON format.
- record timestamp: The timestamp when the event occurred at.
Example of the event data (record value):
{
"eventId": "10d5a05f-7d7b-4cc9-8d05-3816305ac60c",
"occurredAt": "2026-02-17T08:15:22.411886Z",
"eventType": "CONTEXT_DATA_CHANGED",
"source": {
"adminId": "admin",
"sourceType": "adminapp"
},
"data": {
"userId": "4837f519-c128-4540-a6e9-61ec41aa8ac8",
"contextDataChanges": [
{
"surname": {
"newValue": "Muster"
}
}
]
}
}
Authentication and Encryption
The connection to the Kafka broker can be encrypted using TLS and authenticated using different SASL mechanisms.
The Kafka producer's security.protocol property is automatically determined based on the configured TLS and authentication properties in this plugin:
PLAINTEXT: Used when neither TLS nor authentication is configured. Communication is unencrypted and unauthenticated (not recommended for production).SSL: Used when only TLS is configured without authentication. Communication is encrypted. If a keystore is configured, mutual TLS (mTLS) is used for client authentication at the transport layer.SASL_PLAINTEXT: Used when only authentication is configured. Communication is unencrypted (not recommended for production).SASL_SSL: Used when both TLS and authentication are configured. Communication is encrypted. Both mTLS and SASL authentication can be used together in this mode.
Advanced In case a configuration parameter for the producer is missing, it can be added in the advanced settings section.
Note: The Kafka producer's retries property is set to 0.
Any transient failures during event delivery are handled by the service task's own retry mechanism,
ensuring that event delivery is managed consistently within Airlock IAM.
bootstrapServers) List of Kafka bootstrap servers (host:port) used to establish the initial connection to the cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers.
While the order of servers in the list does not matter, it is recommended to list more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka automatically manages and updates connections to the cluster efficiently.
Note on proxy setups:
The Kafka client does not support HTTP/SOCKS proxy configuration, as it uses a native TCP-based protocol and connects directly to brokers. In environments requiring controlled network access, a Kafka-aware gateway or proxy (e.g., Conduktor Gateway) is typically deployed in front of the brokers.
In such setups, this property should point to the gateway endpoint instead of the actual broker addresses. The gateway handles protocol translation, routing, and broker metadata internally. This is different from a traditional web proxy or WAF, which is not compatible with Kafka’s protocol.
For integration, additional logging regarding connections to Kafka can be enabled in the Log4J configuration. An example can be found in the 'Reliable event delivery' section of the IAM documentation.
Related Kafka producer property: bootstrap.servers
topic) Topics are the categories or feed names to which records are published. Kafka topics are divided into partitions, which are the basic unit of parallelism and scalability in Kafka. If the topic does not exist, the broker may automatically create it, depending on its configuration.
key) partition) If specified, all events will be sent to this specific partition. Note that the partition must already exist on the Kafka broker. Otherwise, event delivery will fail. This differs from the topic, which may be created automatically by the broker.
If not configured, the Kafka producer will use its default partitioning strategy, which chooses the partition based on the hash of the configured 'Record Key' property.
authentication) The following authentication mechanisms can be configured:
- Kafka Without Authentication: No authentication mechanism is used.
- Kafka SASL Plain (Username/Password) Authentication: A simple username/password authentication mechanism. Comparable to basic authentication.
- Kafka SASL SCRAM Authentication: A challenge-response authentication mechanism using SCRAM-SHA-256 or SCRAM-SHA-512.
- Kafka SASL OAuth 2.0 Client Credential Grant Authentication A mechanism that uses OAuth 2.0 access tokens for authentication.
For more information, see Authentication using SASL.
Related Kafka producer property: sasl.mechanism
trustStorePath) Related Kafka producer property: ssl.truststore.location
trustStoreType) Related Kafka producer property: ssl.truststore.type
trustStorePassword) Related Kafka producer property: ssl.truststore.password
keyStorePath) Related Kafka producer property: ssl.keystore.location
keyStoreType) Related Kafka producer property: ssl.keystore.type
keyStorePassword) Related Kafka producer property: ssl.keystore.password
keyPassword) Related Kafka producer property: ssl.key.password
clientId) The client ID is included in requests sent to the broker and is used for logging, metrics, and monitoring The value is not used for authentication or authorization and is not stored in Kafka records.
Setting a meaningful client ID is strongly recommended in production environments to improve observability and operational debugging.Related Kafka producer property: client.id
maxBatchProcessingDurationMs) This threshold determines how long the connector will wait for a batch of events to be sent to Kafka before considering the operation as potentially problematic. This helps prevent indefinite blocking and allows for better monitoring of slow Kafka operations.
customProperties) This map allows configuring arbitrary Kafka properties that are not explicitly exposed as standard properties. Existing properties can be overwritten. This is useful for advanced tuning or for setting properties required by custom Kafka interceptors or SASL mechanisms.
Example: Setting linger.ms to 50 or compression.type to gzip.
A list of possible config parameters can be found in the official Kafka Producer Configs documentation.
type: KafkaMessageBrokerConnector
id: KafkaMessageBrokerConnector-xxxxxx
displayName:
comment:
properties:
authentication:
bootstrapServers:
clientId: airlock-iam.event-producer
customProperties:
key: airlock-iam.events
keyPassword:
keyStorePassword:
keyStorePath:
keyStoreType:
maxBatchProcessingDurationMs: 5000
partition:
topic:
trustStorePassword:
trustStorePath:
trustStoreType: