Prometheus metrics and logs

In the following sections, you will find detailed information about the available Prometheus metrics, access log, and application log for Airlock Microgateway. For a convenient way to review metrics and logs, we recommend using our Grafana dashboards. You may also be interested in how logs can suport you during the troubleshooting of your Airlock Microgateway configuration.

Available metrics in Prometheus format

This article lists exposed metrics, i.e., for monitoring licensed throughput and common indicator values. Prometheus supports different metric types such as counter, gauge, histogram, and summary.

Engine metrics

Relabelling by PodMonitor

The following labels are set on all Envoy and Airlock Microgateway Engine metrics when a PodMonitor is configured as described in Using Microgateway Prometheus metrics.

  • managed_by holding the name of the Operator managing the Airlock Microgateway Engine in question.
  • gateway_kind holding the gateway resource kind, i.e. Gateway or SidecarGateway.
  • gateway holding the name of the gateway resource.

Metric name

Metric type

Description

Standard Envoy metrics – see Envoy documentation.

–

Note that metrics are named statistics in the Envoy online documentation:

microgateway_build_info

gauge

Metric shows the static value 1, labeled by the Microgateway Engine version information.

microgateway_authenticated_requests_total

counter

Number of authenticated requests.

The metric has the following labels:

  • authenication_method the authentication method used to authenticate the request
  • authorized a flag indicating if the request was authorized
  • envoy_cluster_name the name of the envoy cluster the request was sent to
  • policy the access control policy that applied to the request
  • upstream the upstream the request was sent to

microgateway_http_downstream_rq_threats_blocked_total

counter

Number of requests (rq) blocked with threatHandlingMode: Block by an Engine filter. Blocks are categorized into block types.

The metric has the following labels:

  • block_type and block_subtype
  • envoy_cluster_name
  • upstream

Example with deny rule-based blocked metric sidecar-based mode:

 
Example
microgateway_http_downstream_rq_threats_blocked_total{block_type="deny_rules", block_subtype="XSS", envoy_cluster_name="upstream_8080", upstream="8080"}

A list of all available rule keys for deny rules can be found in the CR DenyRules reference documentation. See also article CR DenyRules for more information.

Example with a limits-based blocked metric sidecarless mode:

 
Example
microgateway_http_downstream_rq_threats_blocked_total{block_type="limits", block_subtype="parameter_body_size", envoy_cluster_name="httproute/your-backend/backend-tls/rule/0", upstream="your-backend/backend-tls"}

The article CR Limits provides further information for limit-based blocks and potential blocks (in LogOnly mode).

microgateway_http_downstream_rq_threats_logged_total

counter

Number of (rq) logs with threatHandlingMode: LogOnly. All potential blocks are counted. A single request can trigger more than one potential block.

The metric has the following labels:

  • block_type and block_subtype
  • envoy_cluster_name

Example with deny rule-based potential logged metric (sidecar-based mode):

 
Example
microgateway_http_downstream_rq_threats_logged_total{block_type="deny_rules", block_subtype="XSS", envoy_cluster_name="upstream_8080", upstream="8080"}

A list of all available rule keys for deny rules can be found in the CR DenyRules reference documentation. See also article CR DenyRules for more information.

Example with a limits-based logged metric (sidecarless mode):

 
Example
microgateway_http_downstream_rq_threats_logged_total{block_type="limits", block_subtype="bodySize", envoy_cluster_name="httproute/your-backend/backend-tls/rule/0", upstream="your-backend/backend-tls"}

The article CR Limits provides further information for limit-based blocks and potential blocks (in LogOnly mode).

microgateway_license_http_rq_total

counter

Number of total requests.

microgateway_oidc_backchannel_requests_total

counter

Number of backchannel requests sent to OpenID Connect Providers in the context of authentication.

The metric has the following labels:

  • back_channel_cluster_name the name of the envoy cluster the backchannel request was sent to
  • envoy_cluster_name the name of the envoy cluster to which the request was sent that caused the backchannel request
  • policy the access control policy that applied to the request that caused the backchannel request
  • request the type of the backchannel request
  • result the result of the backchannel request
  • upstream the upstream of the request that caused the backchannel request

microgateway_rq_processing_time_ms

histogram

Time in milliseconds between the first received downstream byte and the last sent upstream byte of a request.

The metric has the following labels:

  • envoy_cluster_name
  • upstream

microgateway_rs_processing_time_ms

histogram

Time in milliseconds between the first received upstream byte and the last sent downstream byte of a response.

The metric has the following labels:

  • envoy_cluster_name
  • upstream

microgateway_token_exchanges_total

histogram

Number of performed token exchanges.

The metric has the following labels:

  • envoy_cluster_name the name of the envoy cluster to which the request was sent that caused the token exchange
  • policy the access control policy that applied to the request that caused the token exchange
  • result the result of the token exchange
  • token_exchange_cluster_name the name of the envoy cluster the token exchange request was sent to
  • upstream the upstream of the request that caused the token exchange

microgateway_upstream_responsiveness_ms

histogram

Time in milliseconds between the last request byte sent upstream to the first response byte received from upstream.
If the calculated value is negative, e.g. for backends that are not waiting for requests to be completed, the metric 0 is displayed.

The metric has the following labels:

  • envoy_cluster_name
  • upstream

Operator metrics

Metric name

Metric type

Description

microgateway_build_info

gauge

Metric shows the static value 1, labeled by the Migrogateway Operator version information.

Example:
microgateway_build_info{version=“4.8.0”} 1

microgateway_sidecars

gauge

The number of injected Airlock Microgateway Engine sidecars managed by this Operator instance.

microgateway_license_info1

gauge

Metric shows the static value 1, labeled with the ID of the license that is assigned to the Microgateway Operator. If no license is configured the label value for the label id is set to <MISSING>.

microgateway_license_max_sidecars 1

gauge

The licensed sidecar count. A value of inf means there is no limit.

microgateway_license_valid 1

gauge

Whether the Airlock Microgateway Operator has a valid license 1 or not 0.

microgateway_license_max_rq_count_per_month 1

gauge

The licensed request count per month. When using the community edition license, no monthly count is available, but instead the value NaN is shown.

microgateway_license_is_premium 1

gauge

Whether the active license (if any) is a premium license 1 or not 0.

microgateway_license_expiry_timestamp_seconds 1

gauge

(Unix) timestamp when the active license expires.

microgateway_config_resource_condition

gauge

Whether a condition of the resources Gateway and SidecarGateway is true 1or false 0.

The metric has the following labels:

  • group the API group of the resource
  • kind the API kind of the resource
  • version the API version of the resource
  • name the name of the resource
  • type the type of the monitored condition
  • exported_namespace the namespace the resource is deployed in

microgateway_config_resource_status

gauge

Whether a status of the resources Gateway and SidecarGateway is true 1or false 0.

The metric has the following labels:

  • group the API group of the resource
  • kind the API kind of the resource
  • version the API version of the resource
  • name the name of the resource
  • status the name of the monitored status
  • exported_namespace the namespace the resource is deployed in
1

License-related metric. See also Configuring and monitoring licenses article.
Note that when the license changes, the metrics are updated and the license status changed is logged. This can take up to 1 minute after the secret is adjusted since Kubernetes updates mounted config maps only periodically.

Access log

All requests handled by Airlock Microgateway Engine are logged in the access log. Each filter (deny rules, header rewrites, limits, etc.) may provide additional information. Some filters have settings like threatHandlingMode or operationalMode, directly influencing how much information is logged. For further information, consider the API reference documentation.

The access log follows the Elastic Common Schema (ECS). The information in this article can be used to look up the meaning of log fields and configure a CR Telemetry based on the listed log fields and command operators.

Log field types:

  • Some log fields show logging output using standard Envoy command operators. See online Envoy command operators documentation for details.
  • Other log fields use Microgateway command operators to show logging output. The corresponding Microgateway command operators are listed in the table.

Log field reference table

Log field

Description

Microgateway command operators
(Envoy command operators not shown)

@timestamp

Timestamp of the logged request.

airlock.access_control

Access control-related logs with authorization and authentication information, depending on your access control configuration.

  • Access control logs split into authenticated and authorized status information.
  • Authentication status true is the prerequisite for policy-based authorization.
  • If authorization fails, see policy configurations in the CR AccessControl. Note that with a requireAny policy, the authorization can be successful even if some claims are not fulfilled.

%ACCESS_CONTROL%

airlock.actions.block

Contains block type and block subtype information, including further details, for blocked requests.

%BLOCK_ACTION%

airlock.actions.header_rewrites

Shows a list of header rewrite actions applied on request or response headers. Requires operationalMode to be set to Integration.

Each item in the list contains the following fields:

  • .request.headers – List of headers that have been rewritten.
  • .request.action – Informs about the rewrite action that has been performed on one or more headers.

  • Example:

    • remove
  • .request.type – Informs about the rule type that has been applied.

  • Example:

    • remove-rule
    • allow-rule
  • .request.applied_rules – Names the header rewrite rules that have been applied.

%HEADER_REWRITES%

airlock.actions.log_only

Contains block types and block subtype information, including further details, for threats logged in LogOnly mode.

%LOG_ONLY_ACTIONS%

airlock.http.request.accept_language

Contains the values of the Accept-Language request header.

airlock.http.response.redirect_url

Contains the redirect URL extracted from the Location response header.

airlock.log_correlation.connection_id

Shows the connection ID respectively stream ID. These IDs allow correlating access and application log entries for in-depth analysis.

Application log example:
“ConnectionId”:<X>, “StreamId”:<Y>

%LOG_CORRELATION%

airlock.log_correlation.stream_id

airlock.summary.action

Summary of the Airlock Microgateway action. Can be allowed, denied or error.

%SUMMARY_ACTION%

airlock.summary.details

Shows a response code with detailed information.

%RESPONSE_CODE_DETAILS%

airlock.summary.flags

The Envoy access log response flags provides additional information about response or connection.

%RESPONSE_FLAGS%

airlock.upstream.destination.ip

Direct remote destination IP address of the upstream.

airlock.upstream.destination.port

Direct remote destination port of the upstream.

airlock.upstream.http.version

Shows the HTTP version (1.1/2) of the upstream traffic.

%UPSTREAM_HTTP_VERSION%

destination.ip

Direct destination IP address of the Pod containing the Microgateway Engine sidecar.

destination.port

Direct destination port of the Pod containing the Microgateway Engine sidecar.

ecs.version

Our logging is built on a version of the Elastic Common Schema (ECS) reference.

event.category

ECS categorization field.
For the access log, event.category is always “web”.

event.dataset

Name of the dataset.
For the access log, this is envoy.access.

event.duration

Duration of the request in nanoseconds as recorded by the Microgateway Engine.

%DURATION_IN_NANOSECONDS%

event.end

Time the response was finished by the Microgateway Engine.

%END_TIME(%Y-%m-%dT%T.%3f%z)%

event.kind

ECS categorization field.
For the access log, event.kind is always “event”.

event.module

Module that recorded the event, i.e. envoy.

event.outcome

ECS categorization field.
Indicates the status of the filtering process. When the Microgateway Engine works correctly, the status “success” is shown.
Note that “success” is displayed even if a request was blocked – see also event.type categorization.

event.start

Time the Microgateway Engine received the request.

event.type

ECS categorization field.

Indicates whether a request has been forwarded to the upstream.

Available event types:

  • [ “access”, “denied” ] if a deny rule blocks a request.
  • [ “access”, “allowed” ] if the request was forwarded to the upstream.
  • [ “access”, “error” ], see airlock.summary.details for more information about the error.

%EVENT_TYPE%

http.request.bytes

Total request size in bytes (body and headers).

%REQUEST_HEADERS_AND_BODY_BYTES%

http.request.mime_type

MIME type of the request body.

%REQ_MIME_TYPE%

http.request.body.bytes

Request body size in bytes.

http.request.id

UID of the HTTP request.

http.request.method

HTTP method of the request.

http.request.referrer

Value of the Referer request header.

http.response.body.bytes

Response body size in bytes.

http.response.bytes

Total response size in bytes (body and headers).

%RESPONSE_HEADERS_AND_BODY_BYTES%

http.response.mime_type

MIME type of the response body.

%RESP_MIME_TYPE%

http.response.status_code

HTTP response code.

http.version

Shows the HTTP version (1.1/2) of the application's downstream traffic.

%HTTP_VERSION%

log.level

“info” for all access log entries.

log.logger

“access” for all access log entries.

network.forwarded_ip

Shows the identified remote IP address.

observer.product

Airlock Microgateway

observer.type

The type of observer that created the log entry, i.e. “waap” for Web Application and API Protection.

observer.vendor

Ergon Informatik AG

observer.version

Release version of the Airlock Microgateway, i.e. 4.0.0.

%ENVIRONMENT(ENGINE_VERSION)%

source.ip

Direct remote source IP address of the downstream, as detected by the Microgateway Engine.

source.port

Direct remote source port of the downstream, as detected by the Microgateway Engine.

url.domain

Domain/IP without port number as extracted from the host header.

%HTTP_HOST%

url.path

Path of the request without query string.

Example:
“/my-webapp”

%REQ_QUERY(:PATH)%

url.query

Query string of the request. Note that the ? is excluded from the string.

user_agent.original

Unparsed user agent string.

Example:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36

Airlock Microgateway-specific response code details

The following response code details can appear in the airlock.summary.details log field:

Response code details

Description

airlock_oidc_authentication_redirect

Redirect response when redirected to the OIDC authorization endpoint/login page.

airlock_request_blocked_deny_rule

A deny rule detected malicious data. The request was blocked.

airlock_request_blocked_encoding

The request contains data with invalid encoding (UTF-8 required). The request was blocked.

airlock_request_blocked_graphql

GraphQL schema violations. The request was blocked.

airlock_request_blocked_limit

A limit was exceeded. The request was blocked.

airlock_request_blocked_openapi

OpenAPI validation error. The request was blocked.

airlock_request_blocked_parser

The request contains data that could not be parsed, for example due to a syntax error in JSON data. The request was blocked.

airlock_request_internal_server_error

There was a problem processing the request. Consult the log files for more information.

airlock_request_rejected_openapi

OpenAPI configuration is invalid. The request was rejected.

airlock_response_blocked_openapi

OpenAPI validation error. The response was blocked.

airlock_unauthorized

The applied access control policy has been violated. The request was blocked.

Application log

The Airlock Microgateway Operator, Engine and Session Agent provide an application log which are intended for complex or in-depth analysis. They are usually not required in day-to-day operations, where the access logs are typically sufficient.

If needed, the log level for each component can be adjusted. Note that any change applies to the entire component, not just part of its configuration:

  • Operator: Configure via the Microgateway Helm Chart in the values.yaml using operator.config.logLevel.
  • Engine: Configure via CR GatewayParameters using spec.logging.level.
  • Session Agent: Configure via CR SessionHandling using spec.agent.logging.level.