Metrics, logs and tracing

In the following sections, you will find detailed information about the available metrics, access log, application log, and tracing for Airlock Microgateway. For a convenient way to review metrics and logs, we recommend using our Grafana dashboards. You may also be interested in how logs can suport you during the troubleshooting of your Airlock Microgateway configuration.

Metrics

This article lists exposed metrics, i.e., for monitoring licensed throughput and common indicator values. Prometheus supports different metric types such as counter, gauge, histogram, and summary.

Engine metrics

The following labels are set on all metrics when a PodMonitor and ServiceMonitor is configured as described in Grafana dashboards for metrics and log visualization.

  • managed_by holding the name of the Operator managing the Airlock Microgateway Engine in question.
  • gateway holding the name of the gateway resource.

Metric name

Metric type

Description

Standard Envoy metrics – see Envoy documentation.

–

Note that metrics are named statistics in the Envoy online documentation:

microgateway_build_info

gauge

Metric shows the static value 1, labeled by the Microgateway Engine version information.

microgateway_authenticated_requests_total

counter

Number of authenticated requests.

The metric has the following labels:

  • authenication_method the authentication method used to authenticate the request
  • authorized a flag indicating if the request was authorized
  • envoy_cluster_name the name of the envoy cluster the request was sent to
  • policy the access control policy that applied to the request
  • upstream the upstream the request was sent to

microgateway_http_downstream_rq_threats_blocked_total

counter

Number of requests (rq) blocked with threatHandlingMode: Block by an Engine filter. Blocks are categorized into block types.

The metric has the following labels:

  • block_type and block_subtype
  • envoy_cluster_name
  • upstream

microgateway_http_downstream_rq_threats_logged_total

counter

Number of (rq) logs with threatHandlingMode: LogOnly. All potential blocks are counted. A single request can trigger more than one potential block.

The metric has the following labels:

  • block_type and block_subtype
  • envoy_cluster_name

microgateway_license_http_rq_total

counter

Number of total requests.

microgateway_oidc_backchannel_requests_total

counter

Number of backchannel requests sent to OpenID Connect Providers in the context of authentication.

The metric has the following labels:

  • back_channel_cluster_name the name of the envoy cluster the backchannel request was sent to
  • envoy_cluster_name the name of the envoy cluster to which the request was sent that caused the backchannel request
  • policy the access control policy that applied to the request that caused the backchannel request
  • request the type of the backchannel request
  • result the result of the backchannel request
  • upstream the upstream of the request that caused the backchannel request

microgateway_rq_processing_time_ms

histogram

Time in milliseconds between the first received downstream byte and the last sent upstream byte of a request.

The metric has the following labels:

  • envoy_cluster_name
  • upstream

microgateway_rs_processing_time_ms

histogram

Time in milliseconds between the first received upstream byte and the last sent downstream byte of a response.

The metric has the following labels:

  • envoy_cluster_name
  • upstream

microgateway_token_exchanges_total

histogram

Number of performed token exchanges.

The metric has the following labels:

  • envoy_cluster_name the name of the envoy cluster to which the request was sent that caused the token exchange
  • policy the access control policy that applied to the request that caused the token exchange
  • result the result of the token exchange
  • token_exchange_cluster_name the name of the envoy cluster the token exchange request was sent to
  • upstream the upstream of the request that caused the token exchange

microgateway_upstream_responsiveness_ms

histogram

Time in milliseconds between the last request byte sent upstream to the first response byte received from upstream.
If the calculated value is negative, e.g. for backends that are not waiting for requests to be completed, the metric 0 is displayed.

The metric has the following labels:

  • envoy_cluster_name
  • upstream

Operator metrics

Metric name

Metric type

Description

microgateway_build_info

gauge

Metric shows the static value 1, labeled by the Migrogateway Operator version information.

microgateway_license_info

gauge

Metric shows the static value 1, labeled with the ID of the license that is assigned to the Microgateway Operator. If no license is configured the label value for the label id is set to <MISSING>.

microgateway_license_valid

gauge

Whether the Airlock Microgateway Operator has a valid license 1 or not 0.

microgateway_license_max_rq_count_per_month

gauge

The licensed request count per month. When using the community edition license, no monthly count is available, but instead the value NaN is shown.

microgateway_license_is_premium

gauge

Whether the active license (if any) is a premium license 1 or not 0.

microgateway_license_expiry_timestamp_seconds

gauge

(Unix) timestamp when the active license expires.

microgateway_config_resource_condition

gauge

Whether a condition of the resources Gateway is true 1or false 0.

The metric has the following labels:

  • group the API group of the resource
  • kind the API kind of the resource
  • version the API version of the resource
  • name the name of the resource
  • type the type of the monitored condition
  • exported_namespace the namespace the resource is deployed in

microgateway_config_resource_status

gauge

Whether a status of the resources Gateway is true 1or false 0.

The metric has the following labels:

  • group the API group of the resource
  • kind the API kind of the resource
  • version the API version of the resource
  • name the name of the resource
  • status the name of the monitored status
  • exported_namespace the namespace the resource is deployed in

Access log

All requests handled by Airlock Microgateway Engine are logged in the access log. Each filter (deny rules, header rewrites, limits, etc.) may provide additional information. Some filters have settings like threatHandlingMode or operationalMode, directly influencing how much information is logged. For further information, consider the API reference documentation.

The access log follows the Elastic Common Schema (ECS). The information in this article can be used to look up the meaning of log fields and configure a CR Telemetry based on the listed log fields and command operators.

Log field types:

  • Some log fields show logging output using standard Envoy command operators. See online Envoy command operators documentation for details.
  • Other log fields use Microgateway command operators to show logging output. The corresponding Microgateway command operators are listed in the table.

Log field reference table

Log field

Description

Microgateway command operators
(Envoy command operators not shown)

@timestamp

Timestamp of the logged request.

airlock.access_control

Access control-related logs with authorization and authentication information, depending on your access control configuration.

  • Access control logs split into authenticated and authorized status information.
  • Authentication status true is the prerequisite for policy-based authorization.
  • If authorization fails, see policy configurations in the CR AccessControlPolicy. Note that with a requireAny policy, the authorization can be successful even if some claims are not fulfilled.

%ACCESS_CONTROL%

airlock.actions.block

Contains block type and block subtype information, including further details, for blocked requests.

%BLOCK_ACTION%

airlock.actions.header_rewrites

Shows a list of header rewrite actions applied on request or response headers. Requires operationalMode to be set to Integration.

Each item in the list contains the following fields:

  • .request.headers – List of headers that have been rewritten.
  • .request.action – Informs about the rewrite action that has been performed on one or more headers.
  • .request.type – Informs about the rule type that has been applied.
  • .request.applied_rules – Names the header rewrite rules that have been applied.

%HEADER_REWRITES%

airlock.actions.log_only

Contains block types and block subtype information, including further details, for threats logged in LogOnly mode.

%LOG_ONLY_ACTIONS%

airlock.http.request.accept_language

Contains the values of the Accept-Language request header.

airlock.http.response.redirect_url

Contains the redirect URL extracted from the Location response header.

airlock.http.session.id

ID of the session the request belongs to.

%SESSION_ID%

airlock.log_correlation.connection_id

Shows the connection ID respectively stream ID. These IDs allow correlating access and application log entries for in-depth analysis.

%LOG_CORRELATION%

airlock.log_correlation.stream_id

airlock.summary.action

Summary of the Airlock Microgateway action. Can be allowed, denied or error.

%SUMMARY_ACTION%

airlock.summary.details

Shows a response code with detailed information.

%RESPONSE_CODE_DETAILS%

airlock.summary.flags

The Envoy access log response flags provides additional information about response or connection.

%RESPONSE_FLAGS%

airlock.upstream.destination.ip

Direct remote destination IP address of the upstream.

airlock.upstream.destination.port

Direct remote destination port of the upstream.

airlock.upstream.http.version

Shows the HTTP version (1.1/2) of the upstream traffic.

%UPSTREAM_HTTP_VERSION%

destination.ip

Direct destination IP address of the Pod containing the Microgateway Engine.

destination.port

Direct destination port of the Pod containing the Microgateway Engine.

ecs.version

Our logging is built on a version of the Elastic Common Schema (ECS) reference.

event.category

ECS categorization field.
For the access log, event.category is always “web”.

event.dataset

Name of the dataset.
For the access log, this is envoy.access.

event.duration

Duration of the request in nanoseconds as recorded by the Microgateway Engine.

%DURATION_IN_NANOSECONDS%

event.end

Time the response was finished by the Microgateway Engine.

%END_TIME(%Y-%m-%dT%T.%3f%z)%

event.kind

ECS categorization field.
For the access log, event.kind is always “event”.

event.module

Module that recorded the event, i.e. envoy.

event.outcome

ECS categorization field.
Indicates the status of the filtering process. When the Microgateway Engine works correctly, the status “success” is shown.
Note that “success” is displayed even if a request was blocked – see also event.type categorization.

event.start

Time the Microgateway Engine received the request.

event.type

ECS categorization field.

Indicates whether a request has been forwarded to the upstream.

Available event types:

  • [ “access”, “denied” ] if a deny rule blocks a request.
  • [ “access”, “allowed” ] if the request was forwarded to the upstream.
  • [ “access”, “error” ], see airlock.summary.details for more information about the error.

%EVENT_TYPE%

http.request.bytes

Total request size in bytes (body and headers).

%REQUEST_HEADERS_AND_BODY_BYTES%

http.request.mime_type

MIME type of the request body.

%REQ_MIME_TYPE%

http.request.body.bytes

Request body size in bytes.

http.request.id

UID of the HTTP request.

http.request.method

HTTP method of the request.

http.request.referrer

Value of the Referer request header.

http.response.body.bytes

Response body size in bytes.

http.response.bytes

Total response size in bytes (body and headers).

%RESPONSE_HEADERS_AND_BODY_BYTES%

http.response.mime_type

MIME type of the response body.

%RESP_MIME_TYPE%

http.response.status_code

HTTP response code.

http.version

Shows the HTTP version (1.1/2) of the application's downstream traffic.

%HTTP_VERSION%

log.level

“info” for all access log entries.

log.logger

“access” for all access log entries.

network.forwarded_ip

Shows the identified remote IP address.

observer.product

Airlock Microgateway

observer.type

The type of observer that created the log entry, i.e. “waap” for Web Application and API Protection.

observer.vendor

Ergon Informatik AG

observer.version

Release version of the Airlock Microgateway, i.e. 4.0.0.

%ENVIRONMENT(ENGINE_VERSION)%

source.ip

Direct remote source IP address of the downstream, as detected by the Microgateway Engine.

source.port

Direct remote source port of the downstream, as detected by the Microgateway Engine.

url.domain

Domain/IP without port number as extracted from the host header.

%HTTP_HOST%

url.path

Path of the request without query string.

%REQ_QUERY(:PATH)%

url.query

Query string of the request. Note that the ? is excluded from the string.

user_agent.original

Unparsed user agent string.

Microgateway response code details

The following response code details can appear in the airlock.summary.details log field:

Response code details

Description

airlock_custom_response_upstream_replaced

Upstream response was replaced with a custom response.

airlock_icap_error

There was a problem during ICAP processing.

airlock_icap_response

Response was generated by an ICAP service.

airlock_oidc_authentication_redirect

Redirect response when redirected to the OIDC authorization endpoint/login page.

airlock_oidc_config_race_redirect

Redirect to the original URL from the callback path because the configuration changed.

airlock_oidc_logged_in_redirect

Redirect to the original URL after successful OIDC authentication.

airlock_oidc_race_redirect

Redirect to the original URL from the callback path because the session is already authenticated.

airlock_oidc_signout_redirect

Redirect response after a successful OIDC logout.

airlock_oidc_step_up_redirect

Redirect response when redirected to the OIDC authorization endpoint/login page for step up authorization.

airlock_oidc_try_again_redirect

Redirect to the original URL if the OIDC state parameter was invalid (e.g., state has expired or session has changed).

airlock_request_blocked_csrf

Cross-site request forgery (CSRF) detected. The request was blocked.

airlock_request_blocked_authority

The request contains multiple 'Host' or ':authority' headers. The request was blocked.

airlock_request_blocked_deny_rule

A deny rule detected malicious data. The request was blocked.

airlock_request_blocked_dos_threshold

Denial of Service mitigation was triggered. The request was blocked.

airlock_request_blocked_encoding

The request contains data with invalid encoding (UTF-8 required). The request was blocked.

airlock_request_blocked_graphql

GraphQL schema violation detected. The request was blocked.

airlock_request_blocked_limit

A limit was exceeded. The request was blocked.

airlock_request_blocked_openapi

OpenAPI request validation failed. The request was blocked.

airlock_request_blocked_parser

The request contains data that could not be parsed, for example, due to a syntax error in JSON data. The request was blocked.

airlock_request_internal_server_error

There was a problem processing the request. Consult the log files for more information.

airlock_request_rejected_graphql

GraphQL configuration is invalid. The request was rejected.

airlock_request_rejected_jwt

JWT configuration is invalid. The request was rejected.

airlock_request_rejected_oidc

OIDC configuration is invalid. The request was rejected.

airlock_request_rejected_openapi

OpenAPI configuration is invalid. The request was rejected.

airlock_request_unauthorized_access_control

The applied access control policy has been violated. The request was blocked.

airlock_request_unauthorized_jwt

The JWT extraction or validation has failed. The request was blocked.

airlock_request_unauthorized_oidc

OIDC authentication failed or authorization requirements were not fulfilled. The request was blocked.

airlock_request_unauthorized_token_exchange

OAuth Token Exchange has failed. The request was blocked.

airlock_response_blocked_openapi

OpenAPI response validation failed. The response was blocked.

airlock_session_error

There was a problem concerning the session handling. The request was blocked.

Application log

The Airlock Microgateway Operator, Engine and Session Agent provide an application log which are intended for complex or in-depth analysis. They are usually not required in day-to-day operations, where the access logs are typically sufficient.

If needed, the log level for each component can be adjusted. Note that any change applies to the entire component, not just part of its configuration:

  • Operator: Configure via the Microgateway Helm Chart in the values.yaml using operator.config.logLevel.
  • Engine: Configure via CR GatewayParameters using spec.logging.level.
  • Session Agent: Configure via CR SessionHandling using spec.agent.logging.level.

Tracing

Tracing can be configured to export sampled tracing data to an exporter. See Tracing for more information.