Metrics, Logs and Tracing

In the following sections, you will find detailed information about the available metrics, access log, application log, and tracing for Airlock Microgateway. For a convenient way to review metrics and logs, we recommend using our Grafana dashboards. You may also be interested in how logs can suport you during the troubleshooting of your Airlock Microgateway configuration.

Metrics

This article lists exposed metrics, i.e., for monitoring licensed throughput and common indicator values. Prometheus supports different metric types such as counter, gauge, histogram, and summary.

Engine metrics

The following labels are set on all metrics when a PodMonitor and ServiceMonitor is configured as described in Grafana Dashboards for Metrics and Log Visualization.

  • managed_by holding the name of the Operator managing the Airlock Microgateway Engine in question.
  • gateway holding the name of the gateway resource.
Metric name Metric type Description
Standard Envoy metrics – see Envoy documentation. – Note that metrics are named statistics in the Envoy online documentation:
microgateway_build_info gauge Metric shows the static value 1, labeled by the Microgateway Engine version information.
microgateway_authenticated_requests_total counter Number of authenticated requests.
The metric has the following labels:
  • authenication_method the authentication method used to authenticate the request
  • authorized a flag indicating if the request was authorized
  • envoy_cluster_name the name of the envoy cluster the request was sent to
  • policy the access control policy that applied to the request
  • envoy_virtual_host the virtual host that handled the request
  • envoy_route the route that handled the request
  • country_iso_code country code derived from the request’s configured remote IP address.
microgateway_http_downstream_rq_threats_blocked_total counter Number of requests (rq) blocked with threatHandlingMode: Block by an Engine filter. Blocks are categorized into block types.
The metric has the following labels:
  • block_type and block_subtype
  • envoy_cluster_name
  • envoy_virtual_host the virtual host that handled the request
  • envoy_routethe route that handled the request
  • country_iso_code country code derived from the request’s configured remote IP address.
microgateway_http_downstream_rq_threats_logged_total counter Number of (rq) logs with threatHandlingMode: LogOnly. All potential blocks are counted. A single request can trigger more than one potential block.
The metric has the following labels:
  • block_type and block_subtype
  • envoy_cluster_name
  • envoy_virtual_host the virtual host that handled the request
  • envoy_route the route that handled the request
  • country_iso_code country code derived from the request’s configured remote IP address.
microgateway_license_http_rq_total counter Number of total requests. The metric has the following label:
  • country_iso_code country code derived from the request’s configured remote IP address.
microgateway_oidc_backchannel_requests_total counter Number of backchannel requests sent to OpenID Connect Providers in the context of authentication.
The metric has the following labels:
  • back_channel_cluster_name the name of the envoy cluster the backchannel request was sent to
  • envoy_cluster_name the name of the envoy cluster to which the request was sent that caused the backchannel request
  • policy the access control policy that applied to the request that caused the backchannel request
  • request the type of the backchannel request
  • result the result of the backchannel request
  • envoy_virtual_host the virtual host that handled the request
  • envoy_route the route that handled the request
microgateway_rq_processing_time_ms histogram Time in milliseconds between the first received downstream byte and the last sent upstream byte of a request.
The metric has the following labels:
  • envoy_cluster_name
  • envoy_virtual_host the virtual host that handled the request
  • envoy_route the route that handled the request
microgateway_rs_processing_time_ms histogram Time in milliseconds between the first received upstream byte and the last sent downstream byte of a response.
The metric has the following labels:
  • envoy_cluster_name
  • envoy_virtual_host the virtual host that handled the request
  • envoy_routethe route that handled the request
microgateway_token_exchanges_total histogram Number of performed token exchanges.
The metric has the following labels:
  • envoy_cluster_name the name of the envoy cluster to which the request was sent that caused the token exchange
  • policy the access control policy that applied to the request that caused the token exchange
  • result the result of the token exchange
  • token_exchange_cluster_name the name of the envoy cluster the token exchange request was sent to
  • envoy_virtual_host the virtual host that handled the request
  • envoy_route the route that handled the request
microgateway_upstream_responsiveness_ms histogram Time in milliseconds between the last request byte sent upstream to the first response byte received from upstream.
If the calculated value is negative, e.g. for backends that are not waiting for requests to be completed, the metric 0 is displayed.
The metric has the following labels:
  • envoy_cluster_name
  • envoy_virtual_host the virtual host that handled the request
  • envoy_routethe route that handled the request

Country information

Metrics that provide country information expose the country_iso_code label.

When a country code is available, the label contains that code, for example CH. When no country code is available, Microgateway uses country_iso_code="ZZ" as a placeholder.

ISO 3166-1 makes ZZ available as a user-assigned code. Its use here to indicate unavailable country information is defined by Microgateway; it is not a universal meaning assigned by ISO.

For details on user-assigned codes, see the ISO 3166 glossary.

For the lookup prerequisites and database configuration, see Geolocation.

Session Agent metrics

The following labels are set on all metrics when a PodMonitor and ServiceMonitor is configured as described in Grafana Dashboards for Metrics and Log Visualization.

  • managed_by holding the name of the Operator managing the Airlock Microgateway Engine in question.
  • gateway holding the name of the gateway resource.
Metric name Metric type Description
microgateway_session_store_connection_state gauge Metric shows the current connection state to the session store: 1 denotes working connections, 0 indicates connection problems to the session store.

Operator metrics

Metric name Metric type Description
microgateway_build_info gauge Metric shows the static value 1, labeled by the Migrogateway Operator version information.
microgateway_license_info gauge Metric shows the static value 1, labeled with the ID of the license that is assigned to the Microgateway Operator. If no license is configured the label value for the label id is set to <MISSING>.
microgateway_license_valid gauge Whether the Airlock Microgateway Operator has a valid license 1 or not 0.
microgateway_license_max_rq_count_per_month gauge The licensed request count per month. Without a license or when using an evaluation license, no monthly count is available, but instead the value NaN is shown.
microgateway_license_is_premium gauge Whether the active license (if any) is a premium license 1 or not 0.
microgateway_license_expiry_timestamp_seconds gauge (Unix) timestamp when the active license expires.
microgateway_config_resource_condition gauge Whether a condition of the resources Gateway is true 1or false 0.
The metric has the following labels:
  • group the API group of the resource
  • kind the API kind of the resource
  • version the API version of the resource
  • name the name of the resource
  • type the type of the monitored condition
  • exported_namespace the namespace the resource is deployed in
microgateway_config_resource_status gauge Whether a status of the resources Gateway is true 1or false 0.
The metric has the following labels:
  • group the API group of the resource
  • kind the API kind of the resource
  • version the API version of the resource
  • name the name of the resource
  • status the name of the monitored status
  • exported_namespace the namespace the resource is deployed in

Access log

All requests handled by Airlock Microgateway Engine are logged in the access log. Each filter (deny rules, header rewrites, limits, etc.) may provide additional information. Some filters have settings like threatHandlingMode or operationalMode, directly influencing how much information is logged. For further information, consider the CRD reference documentation.

The access log follows the Elastic Common Schema (ECS). The information in this article can be used to look up the meaning of log fields and configure a CR Telemetry based on the listed log fields and command operators.

Log field types:

  • Some log fields show logging output using standard Envoy command operators. See online Envoy command operators documentation for details.
  • Other log fields use Microgateway command operators to show logging output. The corresponding Microgateway command operators are listed in the table.

Log field reference table

Log field Description Microgateway command operators
(Envoy command operators not shown)
@timestamp Timestamp of the logged request.
airlock.access_control

Access control-related logs with authorization and authentication information, depending on your access control configuration.

  • Access control logs split into authenticated and authorized status information.
  • Authentication status true is the prerequisite for policy-based authorization.
  • If authorization fails, see policy configurations in the CR AccessControlPolicy. Note that with a requireAny policy, the authorization can be successful even if some claims are not fulfilled.
 %ACCESS_CONTROL%
airlock.actions.block Contains block type and block subtype information, including further details, for blocked requests. %BLOCK_ACTION%
airlock.actions.header_rewrites Shows a list of header rewrite actions applied on request or response headers. Requires operationalMode to be set to Integration.
Each item in the list contains the following fields:
  • .request.headers – List of headers that have been rewritten.
  • .request.action – Informs about the rewrite action that has been performed on one or more headers.
  • .request.type – Informs about the rule type that has been applied.
  • .request.applied_rules – Names the header rewrite rules that have been applied.
 %HEADER_REWRITES%
airlock.actions.log_only Contains block types and block subtype information, including further details, for threats logged in LogOnly mode. %LOG_ONLY_ACTIONS%
airlock.http.request.accept_language Contains the values of the Accept-Language request header.
airlock.http.response.redirect_url Contains the redirect URL extracted from the Location response header.
airlock.http.session.id ID of the session the request belongs to. %SESSION_ID%
airlock.log_correlation.connection_id Shows the connection ID respectively stream ID. These IDs allow correlating access and application log entries for in-depth analysis. %LOG_CORRELATION%
airlock.log_correlation.stream_id
airlock.summary.action Summary of the Airlock Microgateway action. Can be allowed, denied or error. %SUMMARY_ACTION%
airlock.summary.details

Shows a response code with detailed information.

 %RESPONSE_CODE_DETAILS%
airlock.summary.flags The Envoy access log response flags provides additional information about response or connection. %RESPONSE_FLAGS%
airlock.upstream.destination.ip Direct remote destination IP address of the upstream.
airlock.upstream.destination.port Direct remote destination port of the upstream.
airlock.upstream.http.version Shows the HTTP version (1.1/2) of the upstream traffic. %UPSTREAM_HTTP_VERSION%
destination.ip Direct destination IP address of the Pod containing the Microgateway Engine.
destination.port Direct destination port of the Pod containing the Microgateway Engine.
ecs.version Our logging is built on a version of the Elastic Common Schema (ECS) reference.
event.category ECS categorization field.
For the access log, event.category is always “web”.
event.dataset Name of the dataset.
For the access log, this is envoy.access.
event.duration Duration of the request in nanoseconds as recorded by the Microgateway Engine. %DURATION_IN_NANOSECONDS%
event.end Time the response was finished by the Microgateway Engine. %END_TIME(%Y-%m-%dT%T.%3f%z)%
event.kind ECS categorization field.
For the access log, event.kind is always “event”.
event.module Module that recorded the event, i.e. envoy.
event.outcome ECS categorization field.
Indicates the status of the filtering process. When the Microgateway Engine works correctly, the status “success” is shown.
Note that “success” is displayed even if a request was blocked – see also event.type categorization.
event.start Time the Microgateway Engine received the request.
event.type ECS categorization field.
Indicates whether a request has been forwarded to the upstream.
Available event types:
  • [ “access”, “denied” ] if a deny rule blocks a request.
  • [ “access”, “allowed” ] if the request was forwarded to the upstream.
  • [ “access”, “error” ], see airlock.summary.details for more information about the error.
 %EVENT_TYPE%
geo GeoIP information for the request. Depending on the information available in the configured GeoIP database, the object can contain:
  • .city_name – city name
  • .region_iso_code – region ISO code
  • .country_iso_code – country ISO code
    • To verify the address used for the lookup, inspect network.forwarded_ip in the default access log. This field contains the identified remote IP address, which can differ from the direct downstream connection address recorded in source.ip.
 %GEO(city_name)%
%GEO(region_iso_code)%
%GEO(country_iso_code)%
http.request.bytes Total request size in bytes (body and headers). %REQUEST_HEADERS_AND_BODY_BYTES%
http.request.mime_type MIME type of the request body. %REQ_MIME_TYPE%
http.request.body.bytes Request body size in bytes.
http.request.id UID of the HTTP request.
http.request.method HTTP method of the request.
http.request.referrer Value of the Referer request header.
http.response.body.bytes Response body size in bytes.
http.response.bytes Total response size in bytes (body and headers). %RESPONSE_HEADERS_AND_BODY_BYTES%
http.response.mime_type MIME type of the response body. %RESP_MIME_TYPE%
http.response.status_code HTTP response code.
http.version Shows the HTTP version (1.1/2) of the application’s downstream traffic. %HTTP_VERSION%
log.level “info” for all access log entries.
log.logger “access” for all access log entries.
network.forwarded_ip Shows the identified remote IP address.
observer.product Airlock Microgateway
observer.type The type of observer that created the log entry, i.e. “waap” for Web Application and API Protection.
observer.vendor Ergon Informatik AG
observer.version Release version of the Airlock Microgateway, i.e. 4.0.0. %ENVIRONMENT(ENGINE_VERSION)%
source.ip Direct remote source IP address of the downstream, as detected by the Microgateway Engine.
source.port Direct remote source port of the downstream, as detected by the Microgateway Engine.
url.domain Domain/IP without port number as extracted from the host header. %HTTP_HOST%
url.path Path of the request without query string. %REQ_QUERY(:PATH)%
url.query Query string of the request. Note that the ? is excluded from the string.
user_agent.original Unparsed user agent string.

Microgateway response code details

The following response code details can appear in the airlock.summary.details log field:

Response code details Description
airlock_custom_response_upstream_replaced Upstream response was replaced with a custom response.
airlock_icap_error There was a problem during ICAP processing.
airlock_icap_response Response was generated by an ICAP service.
airlock_oidc_authentication_redirect Redirect response when redirected to the OIDC authorization endpoint/login page.
airlock_oidc_config_race_redirect Redirect to the original URL from the callback path because the configuration changed.
airlock_oidc_logged_in_redirect Redirect to the original URL after successful OIDC authentication.
airlock_oidc_race_redirect Redirect to the original URL from the callback path because the session is already authenticated.
airlock_oidc_signout_redirect Redirect response after a successful OIDC logout.
airlock_oidc_step_up_redirect Redirect response when redirected to the OIDC authorization endpoint/login page for step up authorization.
airlock_oidc_try_again_redirect Redirect to the original URL if the OIDC state parameter was invalid (e.g., state has expired or session has changed).
airlock_request_blocked_authority The request contains multiple ‘Host’ or ‘:authority’ headers. The request was blocked.
airlock_request_blocked_csrf Cross-site request forgery (CSRF) detected. The request was blocked.
airlock_request_blocked_deny_rule A deny rule detected malicious data. The request was blocked.
airlock_request_blocked_dos_threshold Denial of Service mitigation was triggered. The request was blocked.
airlock_request_blocked_encoding The request contains data with invalid encoding (UTF-8 required). The request was blocked.
airlock_request_blocked_graphql GraphQL schema violation detected. The request was blocked.
airlock_request_blocked_limit A limit was exceeded. The request was blocked.
airlock_request_blocked_openapi OpenAPI request validation failed. The request was blocked.
airlock_request_blocked_parser The request contains data that could not be parsed, for example, due to a syntax error in JSON data. The request was blocked.
airlock_request_blocked_rate_limit Local rate limit was exceeded. The request was blocked.
airlock_request_internal_server_error There was a problem processing the request. Consult the log files for more information.
airlock_request_rejected_graphql GraphQL configuration is invalid. The request was rejected.
airlock_request_rejected_jwt JWT configuration is invalid. The request was rejected.
airlock_request_rejected_oidc OIDC configuration is invalid. The request was rejected.
airlock_request_rejected_openapi OpenAPI configuration is invalid. The request was rejected.
airlock_request_unauthorized_access_control The applied access control policy has been violated. The request was blocked.
airlock_request_unauthorized_jwt The JWT extraction or validation has failed. The request was blocked.
airlock_request_unauthorized_oidc OIDC authentication failed or authorization requirements were not fulfilled. The request was blocked.
airlock_request_unauthorized_token_exchange OAuth Token Exchange has failed. The request was blocked.
airlock_response_blocked_openapi OpenAPI response validation failed. The response was blocked.
airlock_session_error There was a problem concerning the session handling. The request was blocked.

Application log

The Airlock Microgateway Operator, Engine and Session Agent provide an application log which are intended for complex or in-depth analysis. They are usually not required in day-to-day operations, where the access logs are typically sufficient.

If needed, the log level for each component can be adjusted. Note that any change applies to the entire component, not just part of its configuration:

  • Operator: Configure via the Microgateway Helm Chart in the values.yaml using operator.config.logLevel.
  • Engine: Configure via CR GatewayParameters using spec.logging.level.
  • Session Agent: Configure via CR SessionHandling using spec.agent.logging.level.

Tracing

Tracing can be configured to export sampled tracing data to an exporter. See Tracing for more information.

External links: