Prometheus metrics and access logs

Available metrics in Prometheus format

This article lists exposed metrics, i.e., for monitoring licensed throughput and common indicator values. Prometheus supports different metric types such as counter, gauge, histogram, and summary.

Engine metrics

Relabelling by PodMonitor

The following labels are set on all Envoy and Airlock Microgateway Engine metrics when a PodMonitor is configured as described in Using Microgateway Prometheus metrics.

  • managed_by holding the name of the Operator managing the Airlock Microgateway Engine in question.
  • gateway_kind holding the gateway resource kind, i.e. Gateway or SidecarGateway.
  • gateway holding the name of the gateway resource.

Metric name

Metric type

Description

Standard Envoy metrics – see Envoy documentation.

–

microgateway_build_info

gauge

Metric shows the static value 1, labeled by the Microgateway Engine version information.

microgateway_license_http_rq_total

counter

Number of total requests.

microgateway_http_downstream_rq_threats_blocked_total

counter

Number of requests (rq) blocked with threatHandlingMode: Block by an Engine filter. Blocks are categorized into block types.

  • The metric has the following labels:
  • block_type and block_subtype
  • envoy_cluster_name
  • upstream

Example with deny rule-based blocked metric sidecar-based mode:

microgateway_http_downstream_rq_threats_blocked_total{block_type="deny_rules", block_subtype="XSS", envoy_cluster_name="upstream_8080", upstream="8080"}

A list of all available rule keys for deny rules can be found in the CR DenyRules reference documentation. See also article CR DenyRules for more information.

Example with a limits-based blocked metric sidecarless mode:

microgateway_http_downstream_rq_threats_blocked_total{block_type="limits", block_subtype="parameter_body_size", envoy_cluster_name="httproute/your-backend/backend-tls/rule/0", upstream="your-backend/backend-tls"}

The article CR Limits provides further information for limit-based blocks and potential blocks (in LogOnly mode).

microgateway_http_downstream_rq_threats_logged_total

counter

Number of (rq) logs with threatHandlingMode: LogOnly. All potential blocks are counted. A single request can trigger more than one potential block.

  • The metric has the following labels:
  • block_type and block_subtype
  • envoy_cluster_name

Example with deny rule-based potential logged metric (sidecar-based mode):

microgateway_http_downstream_rq_threats_logged_total{block_type="deny_rules", block_subtype="XSS", envoy_cluster_name="upstream_8080", upstream="8080"}

A list of all available rule keys for deny rules can be found in the CR DenyRules reference documentation. See also article CR DenyRules for more information.

Example with a limits-based logged metric (sidecarless mode):

microgateway_http_downstream_rq_threats_logged_total{block_type="limits", block_subtype="bodySize", envoy_cluster_name="httproute/your-backend/backend-tls/rule/0", upstream="your-backend/backend-tls"}

The article CR Limits provides further information for limit-based blocks and potential blocks (in LogOnly mode).

microgateway_rq_processing_time_ms

histogram

Time in milliseconds between the first received downstream byte and the last sent upstream byte of a request.

  • The metric has the following labels:
  • envoy_cluster_name
  • upstream

microgateway_rs_processing_time_ms

histogram

Time in milliseconds between the first received upstream byte and the last sent downstream byte of a response.

  • The metric has the following labels:
  • envoy_cluster_name
  • upstream

microgateway_upstream_responsiveness_ms

histogram

Time in milliseconds between the last request byte sent upstream to the first response byte received from upstream.
If the calculated value is negative, e.g. for backends that are not waiting for requests to be completed, the metric 0 is displayed.

  • The metric has the following labels:
  • envoy_cluster_name
  • upstream

Operator metrics

Metric name

Metric type

Description

microgateway_build_info

gauge

Metric shows the static value 1, labeled by the Migrogateway Operator version information.

Example:
microgateway_build_info{version="4.4.0"} 1

microgateway_sidecars

gauge

The number of injected Airlock Microgateway Engine sidecars managed by this Operator instance.

microgateway_license_info1

gauge

Metric shows the static value 1, labeled with the ID of the license that is assigned to the Microgateway Operator. If no license is configured the label value for the label id is set to <MISSING>.

microgateway_license_max_sidecars 1

gauge

The licensed sidecar count. A value of inf means there is no limit.

microgateway_license_valid 1

gauge

Whether the Airlock Microgateway Operator has a valid license 1 or not 0.

microgateway_license_max_rq_count_per_month 1

gauge

The licensed request count per month. When using the community edition license, no monthly count is available, but instead the value NaN is shown.

microgateway_license_is_premium 1

gauge

Whether the active license (if any) is a premium license 1 or not 0.

microgateway_license_expiry_timestamp_seconds 1

gauge

(Unix) timestamp when the active license expires.

microgateway_config_resource_condition

gauge

Whether a condition of the resources Gateway and SidecarGateway is true 1or false 0.

  • The metric has the following labels:
  • group the API group of the resource
  • kind the API kind of the resource
  • version the API version of the resource
  • name the name of the resource
  • type the type of the monitored condition
  • exported_namespace the namespace the resource is deployed in

microgateway_config_resource_status

gauge

Whether a status of the resources Gateway and SidecarGateway is true 1or false 0.

  • The metric has the following labels:
  • group the API group of the resource
  • kind the API kind of the resource
  • version the API version of the resource
  • name the name of the resource
  • status the name of the monitored status
  • exported_namespace the namespace the resource is deployed in
1

License-related metric. See also Configuration and monitoring of licenses article.
Note that when the license changes, the metrics are updated and the license status changed is logged. This can take up to 1 minute after the secret is adjusted since Kubernetes updates mounted config maps only periodically.

Access log field reference

The access log follows the Elastic Common Schema (ECS). The information in this article can be used to look up the meaning of log fields and configure a CR Telemetry based on the listed log fields and command operators.

  • Log field types:
  • Some log fields show logging output using standard Envoy command operators. See online Envoy command operators documentation for details.
  • Other log fields use custom command operators to show logging output. The corresponding custom command operators are listed in the table.

Log field reference table

Log field

Description

Custom command operators
(Envoy command operators not shown)

@timestamp

Timestamp of the logged request.

airlock.access_control

Access control-related logs with authorization and authentication information, depending on your access control configuration.

  • Access control logs split into authenticated and authorized status information.
  • Authentication status true is the prerequisite for policy-based authorization.
  • If authorization fails, see policy configurations in the CR AccessControl. Note that with a requireAny policy, the authorization can be successful even if some claims are not fulfilled.

%ACCESS_CONTROL%

airlock.actions.block

Contains block type and block subtype information, including further details, for blocked requests.

%BLOCK_ACTION%

airlock.actions.header_rewrites

Shows a list of header rewrite actions applied on request or response headers. Requires operationalMode to be set to Integration.

  • Each item in the list contains the following fields:
  • .request.headers – List of headers that have been rewritten.
  • .request.action – Informs about the rewrite action that has been performed on one or more headers.
    • Example:
    • remove
  • .request.type – Informs about the rule type that has been applied.
    • Example:
    • remove-rule
    • allow-rule
  • .request.applied_rules – Names the header rewrite rules that have been applied.

%HEADER_REWRITES%

airlock.actions.log_only

Contains block types and block subtype information, including further details, for threats logged in LogOnly mode.

%LOG_ONLY_ACTIONS%

airlock.http.request.accept_language

Contains the values of the Accept-Language request header.

airlock.http.response.redirect_url

Contains the redirect URL extracted from the Location response header.

airlock.log_correlation.connection_id

Shows the connection ID respectively stream ID. These IDs allow correlating access and application log entries for in-depth analysis.

Application log example:
"ConnectionId":<X>, "StreamId":<Y>

%LOG_CORRELATION%

airlock.log_correlation.stream_id

airlock.summary.action

Summary of the Airlock Microgateway action. Can be allowed, blocked or error.

%SUMMARY_ACTION%

airlock.summary.details

Shows a response code with detailed information.

%RESPONSE_CODE_DETAILS%

airlock.summary.flags

The Envoy access log response flags provides additional information about response or connection.

%RESPONSE_FLAGS%

airlock.upstream.destination.ip

Direct remote destination IP address of the upstream.

airlock.upstream.destination.port

Direct remote destination port of the upstream.

airlock.upstream.http.version

Shows the HTTP version (1.1/2) of the upstream traffic.

%UPSTREAM_HTTP_VERSION%

destination.ip

Direct destination IP address of the Pod containing the Microgateway Engine sidecar.

destination.port

Direct destination port of the Pod containing the Microgateway Engine sidecar.

ecs.version

Our logging is built on a version of the Elastic Common Schema (ECS) reference.

event.category

ECS categorization field.
For the access log, event.category is always "web".

event.dataset

Name of the dataset.
For the access log, this is envoy.access.

event.duration

Duration of the request in nanoseconds as recorded by the Microgateway Engine.

%DURATION_IN_NANOSECONDS%

event.end

Time the response was finished by the Microgateway Engine.

%END_TIME(%Y-%m-%dT%T.%3f%z)%

event.kind

ECS categorization field.
For the access log, event.kind is always "event".

event.module

Module that recorded the event, i.e. envoy.

event.outcome

ECS categorization field.
Indicates the status of the filtering process. When the Microgateway Engine works correctly, the status "success" is shown.
Note that "success" is displayed even if a request was blocked – see also event.type categorization.

event.start

Time the Microgateway Engine received the request.

event.type

ECS categorization field.

Indicates whether a request has been forwarded to the upstream.

  • Available event types:
  • [ "access", "denied" ] if a deny rule blocks a request.
  • [ "access", "allowed" ] if the request was forwarded to the upstream.
  • [ "access", "error" ], see airlock.summary.details for more information about the error.

%EVENT_TYPE%

http.request.bytes

Total request size in bytes (body and headers).

%REQUEST_HEADERS_AND_BODY_BYTES%

http.request.mime_type

MIME type of the request body.

%REQ_MIME_TYPE%

http.request.body.bytes

Request body size in bytes.

http.request.id

UID of the HTTP request.

http.request.method

HTTP method of the request.

http.request.referrer

Value of the Referer request header.

http.response.body.bytes

Response body size in bytes.

http.response.bytes

Total response size in bytes (body and headers).

%RESPONSE_HEADERS_AND_BODY_BYTES%

http.response.mime_type

MIME type of the response body.

%RESP_MIME_TYPE%

http.response.status_code

HTTP response code.

http.version

Shows the HTTP version (1.1/2) of the application's downstream traffic.

%HTTP_VERSION%

log.level

"info" for all access log entries.

log.logger

"access" for all access log entries.

  • Other possible values:
  • "application" for Microgateway Engine log entries.

network.forwarded_ip

Shows the identified remote IP address.

observer.product

Airlock Microgateway

observer.type

The type of observer that created the log entry, i.e. "waap" for Web Application and API Protection.

observer.vendor

Ergon Informatik AG

observer.version

Release version of the Airlock Microgateway, i.e. 4.0.0.

%ENVIRONMENT(ENGINE_VERSION)%

source.ip

Direct remote source IP address of the downstream, as detected by the Microgateway Engine.

source.port

Direct remote source port of the downstream, as detected by the Microgateway Engine.

url.domain

Domain/IP without port number as extracted from the host header.

%HTTP_HOST%

url.path

Path of the request without query string.

Example:
"/my-webapp"

%REQ_QUERY(:PATH)%

url.query

Query string of the request. Note that the ? is excluded from the string.

user_agent.original

Unparsed user agent string.

Example:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36

Airlock Microgateway-specific response code details

The following response code details can appear in the airlock.summary.details log field:

Response code details

Description

airlock_oidc_authentication_redirect

Redirect response when redirected to the OIDC authorization endpoint/login page.

airlock_request_blocked_deny_rule

A deny rule detected malicious data. The request was blocked.

airlock_request_blocked_encoding

The request contains data with invalid encoding (UTF-8 required). The request was blocked.

airlock_request_blocked_graphql

GraphQL schema violations. The request was blocked.

airlock_request_blocked_limit

A limit was exceeded. The request was blocked.

airlock_request_blocked_openapi

OpenAPI validation error. The request was blocked.

airlock_request_blocked_parser

The request contains data that could not be parsed, for example due to a syntax error in JSON data. The request was blocked.

airlock_request_internal_server_error

There was a problem processing the request. Consult the log files for more information.

airlock_request_rejected_openapi

OpenAPI configuration is invalid. The request was rejected.

airlock_response_blocked_openapi

OpenAPI validation error. The response was blocked.

airlock_unauthorized

The applied access control policy has been violated. The request was blocked.