Custom Resource Definitions

To configure the settings and aspects of the Airlock Microgateway in sidecar-based and sidecarless data plane mode, we created several Custom Resource Definitions. Detailed configuration information including examples for most CRDs are documented in the Airlock Microgateway API reference documentation.
 

CR AccessControl

The CR AccessControl specifies the options to perform access control with a Microgateway Engine container.

Applicable data plane mode

  • Sidecar-based data plane mode.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

References

This CR specifies the desired identity propagation.

This CR configures client authentication via a JSON Web Token.

This CR specifies how the Airlock Microgateway Engine interacts with an OpenID Provider (OP).

This CR specifies how the Airlock Microgateway Engine interacts with an OAuth Token Exchange endpoint.

Prerequisites

-

About policy configurations:

  • Multiple policies can be configured. The requestConditions work as a policy selector.
  • The list of policies is processed from top to bottom – the first matching policy will be applied.
  • When multiple policies are configured, a fallback policy without requestConditions is required to catch requests that do not match any policy selector. Otherwise, a validation error will appear. The fallback policy must be added at the end of the policy list.
  • If only a single policy is configured without requestConditions, an additional fallback policy is not required.
  • A mix of requireAll and requireAny authorization definition is not supported.

Token introspection:
An introspection strategy (always or random) can be configured to check the validity of non–expired OIDC tokens. We recommend using the strategy always to check the token validity with each request. Note that this strategy consumes more system resources but provides the highest level of security.

Introspection strategy

Microgateway Engine request handling

always

  • If the token introspection endpoint is not reachable, the current request is blocked. The token remains valid.
  • If the token introspection returns that the token is invalid or expired, the token will be deleted. A token refresh or new OIDC flow is required to re–authenticate the user.

random

  • If the token introspection endpoint is not reachable, the introspection attempt is discarded, and the token remains valid with this attempt.
  • If the token introspection returns that the token is invalid or expired, the token will be deleted. A token refresh or new OIDC flow is required to re–authenticate the user.

CR AccessControlPolicy

The CR AccessControlPolicy is a Direct Attached Policy for the K8s Gateway API. It specifies the options to perform access control with an Airlock Microgateway.

Applicable data plane mode

  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

References

spec.targetRefs[].HTTPRoute

Specifies the HTTPRoute the Microgateway policy is attached to.

This CR specifies the desired identity propagation.

This CR configures client authentication via a JSON Web Token.

This CR specifies how the Airlock Microgateway Engine interacts with an OpenID Provider (OP).

This CR specifies how the Airlock Microgateway Engine interacts with an OAuth Token Exchange endpoint.

Prerequisites

-

About policy configurations:

  • Multiple policies can be configured. The requestConditions work as a policy selector.
  • The list of policies is processed from top to bottom – the first matching policy will be applied.
  • When multiple policies are configured, a fallback policy without requestConditions is required to catch requests that do not match any policy selector. Otherwise, a validation error will appear. The fallback policy must be added at the end of the policy list.
  • If only a single policy is configured without requestConditions, an additional fallback policy is not required.
  • A mix of requireAll and requireAny authorization definition is not supported.

Token introspection:
An introspection strategy (always or random) can be configured to check the validity of non–expired OIDC tokens. We recommend using the strategy always to check the token validity with each request. Note that this strategy consumes more system resources but provides the highest level of security.

Introspection strategy

Microgateway Engine request handling

always

  • If the token introspection endpoint is not reachable, the current request is blocked. The token remains valid.
  • If the token introspection returns that the token is invalid or expired, the token will be deleted. A token refresh or new OIDC flow is required to re–authenticate the user.

random

  • If the token introspection endpoint is not reachable, the introspection attempt is discarded, and the token remains valid with this attempt.
  • If the token introspection returns that the token is invalid or expired, the token will be deleted. A token refresh or new OIDC flow is required to re–authenticate the user.

CR APIProtection

The CR APIProtection contains the configuration for API security.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

This CR contains the configuration for the GraphQL specification.

This CR contains the configuration for the OpenAPI specification.

Prerequisites

-

CR ContentSecurity

The CR ContentSecurity specifies the options to secure an upstream web application with a Microgateway Engine container. It does so by referencing various other CRs that cover different aspects of web application security.
If references are not explicitly configured, default settings designed to work with most upstream services will be applied.

Applicable data plane mode

  • Sidecar-based data plane mode.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

References

Configures relevant OpenAPI or GraphQL configuration resource.

Configures Cross Site Request Forgery (CSRF) protection.

Configures request filtering using built–in and custom deny rules.

Configures request and response header manipulations.

Configures various size checks on requests.

Configures content parsers.

Prerequisites

-

 
Notice

Note that the configuration of spec.apiProtection.graphQLRef and spec.apiProtection.OpenAPIRef is deprecated in this CR. Configure policies in CR APIProtection instead.

CR ContentSecurityPolicy

The CR ContentSecurityPolicy is a Direct Attached Policy for the K8s Gateway API. It specifies the options to secure an upstream web application with a Microgateway. It does so by referencing various other CRs that cover different aspects of web application security.

Applicable data plane mode

  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

References

spec.targetRefs[].HTTPRoute

Specifies the HTTPRoute the Microgateway policy is attached to.

Configures relevant OpenAPI or GraphQL configuration resource.

Configures Cross-Site Request Forgery (CSRF) protection.

Configures request filtering using built-in and custom deny rules.

Configures request and response header manipulations.

Configures various size checks on requests.

Configures content parsers.

Prerequisites

-

CR CSRFProtection

The CR CSRFProtection provides the configuration options for cross-site request forgery (CSRF) protection.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

Prerequisites

-

CR DenyRules

The CR DenyRules configures request filtering using Airlock built-in and custom deny rules. Deny rules establish a negative security model. They define prohibited patterns that, when a match is found in a request, lead to it being blocked from reaching the upstream web application.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

Prerequisites

-

Configuration options:

  • The security level of the applied deny rules.
  • The threatHandlingMode (i.e., Block or LogOnly) can be configured globally and on the ruleKey level. Non–deny rule depending threatHandlingMode can be configured on feature level, i.e. Limits, OpenAPI and GraphQL within their CR.
  • Deny rule overrides to change settings (e.g. security level or threat handling mode) of specific deny rules.
  • Deny rule exceptions for requests that match one or more deny rules but should not be blocked. Exceptions can be configured and fine–tuned to reduce the number of false positives using blockedData and/or requestConditions based on various characteristics.
  • The definition of custom deny rules.

Different security levels can be configured on global and deny rule levels. This makes it possible to configure the balanced processing of requests with a high level of security and a low number of false positives simultaneously.

Global level

Explanation

Unfiltered

In level Unfiltered, no deny rules are applied. This means, no requests will be blocked.

Basic

Deny rules in level Basic focus on a low false–positive rate, simplifying the integration of applications. Note, however, that certain attack variants may not be covered.

Indications for using the level Basic:

  • Level Standard requires too many exceptions.
  • The upstream application access is protected by authentication.

Standard

Level Standard is the default setting. It provides strong filters and a low false–positive rate. Exceptions may be required for input fields containing syntactical elements similar to JavaScript or SQL.

Indications for using the level Standard:

  • The application is complex or dynamic.
  • The application uses many input fields with unrestricted input values, e.g., free texts or comments.
  • Application access is protected by upstream authentication.
  • Level Strict requires too many exceptions.

Strict

Level Strict focuses on blocking many potential attack variants. This level is recommended for very sensitive applications and typically requires some integration effort.

Indications for using level Strict:

  • Login and critical pages are exposed directly to the Internet without upstream authentication.
  • The application is rather simple.
  • Application data is very sensitive (high risk).
  • Low code quality of an application.

Override global settings for specific deny rules
For fine–tuning, i.e., to reduce the number of false positives, the security level or the threat handling mode for individual deny rules may be configured differently from the global settings. To do so, configure non–global security levels for deny rules using the ruleKeys or the types element.

 
Notice

The deny rule set is constantly evolving and updated to respond to the latest threats. Each deny rule is identified by a deny rule key that can be referenced for configuration purposes. Each ruleKey refers to different attack types such as SQL injection, XSS, GraphQL, TEMPLATE injection, etc.
To handle possible false positives, lower the security level or define fine-granular deny rule exceptions. If undefined, default settings are applied, designed to work with most upstream web application services.

Deny rule exceptions
Deny rule exceptions can be defined for different data elements.

Data element types for exceptions:

  • blockedData for exceptions on blocked data elements, e.g., for blocked parameters, headers or JSON data.
    Note that blockedData only allows requests if the string triggering the false positive was in the configured parameter, header or JSON data.
  • requestConditions, to create exceptions for general characteristics of blocked requests, e.g. request path or originating remote IP.

Possible deny rule exception configuration options:

  • Define a deny rule exception only with blockedData:
  • This way, the configured exception is applied unconditionally to all paths, HTTP methods, remote IP addresses, ...

  • Define a deny rule exception with blockedData and requestConditions:
  • This is the recommended configuration, as it only applies the exceptions to requests that matches the requestConditions.

  • Define a deny rule exception only with requestConditions:
  • This configuration should be chosen carefully, as deny rule checks are not applied at all to requests which matches the requestConditions.

Note that exceptions will only reduce blocks if the configured exception covers all data elements that trigger deny rules.

For example, if an exception is configured to unblock search parameters, a request containing a search and a language parameter could still be blocked if the language parameter triggers a deny rule. The same mechanism applies to other blocked data elements such as parameters, headers or JSON data.

Deny rule exceptions for JSON content
Body data of requests with content–type header application/json are parsed as JSON content and might cause false positives. Deny rule exceptions can be defined for JSON keys or values based on the access log information. Additionally, exceptions can be restricted to a defined JSONPath. In the JSONPath expressions, wildcards (*) can be used, like in this JSONPath syntax reference.

Example:

 
Example
{ 
  "books": [ 
    { 
      "title": "DevSecOps Starters Guide", 
      "author": "Joe Bloggs", 
      "stars": 723 
    }, 
    { 
      "title": "JSONPath for Geeks", 
      "author": "Jane Smith", 
      "stars": 27 
    } 
  ] 
}

This JSON example contains 6 key–value pairs (aka leaves), 3 pairs for each book. The values of the JSON leaves can be filtered with the corresponding JSONPath from the following table.

 
Example
JSONPath          | Value 
––––––––––––––––––|–––––––––––––––––––––––––––– 
$.books[0].title  | "DevSecOps Starters Guide", 
$.books[0].author | "Joe Bloggs", 
$.books[0].stars  | 723 
$.books[1].title  | "JSONPath for Geeks", 
$.books[1].author | "Jane Smith", 
$.books[1].stars  | 27
 
Info

Additional JSONPath examples:

  • $..author – matches every leaf of the key author (Joe Bloggs and Jane Smith) in every book.
  • $.books[*].author – matches every leaf of the key author in all books.

For instance, an exception for all books of authors named Joe can be configured as follows.

Example:

 
Example
apiVersion: microgateway.airlock.com/v1alpha1
kind: DenyRules
metadata:
  name: my–webapp
 spec: 
  request: 
    builtIn:
      exceptions: 
        – blockedData: 
            json: 
              jsonPath: $.books[*].author 
              value: 
                matcher: 
                  contains: Joe
 
Functional limitation
  • For deny rule exceptions, operators [] with expressions and script expressions are not supported. For example, $..book[?(@.age)].title does NOT work as JSONPath expression.
  • The JSONPath expression must always match a leaf. For example, $.books[*] does not contain a leaf and will not work, whereas $.books[*].author does.

Custom deny rules
Custom deny rules can be configured to complement the built–in deny rules.

Difference to built–in deny rules:

  • Custom rules can be added in the CR DenyRules.
  • Security levels and exceptions cannot be applied to custom deny rules.
 
Info

If one of the configured rules matches a request, the ruleKey is included in the corresponding access log message. Therefore, we recommend using short, descriptive denominations for rule keys.

Note that the ruleKey must be a unique denomination and match ^[A–Z][A–Z0–9_]*$, e.g. ruleKey: MY_CUSTOM_KEY_01.

CR EnvoyCluster

The CR EnvoyCluster is an additional Envoy Cluster resource that is added to those defined by the Airlock Microgateway. The CR allows configuring additional clusters if a native Envoy HTTP filter relies on it. For the CR EnvoyCluster, the Envoy configuration language must be used.
For an example targeting debugging, see article Debugging with Envoy filters.

Applicable data plane mode

  • Sidecar-based data plane mode.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

References

Prerequisites

-

When do I have to configure an CR EnvoyCluster?

  • Envoy HTTP filters (CR EnvoyHTTPFilter) that connect to an upstream service require configuring the corresponding cluster with EnvoyCluster.
  • Envoy HTTP filters that do their work without an external cluster do not need an EnvoyCluster configured.
 
Notice
  • You should use the CRs EnvoyHTTPFilter and EnvoyClusters only if a feature is unavailable in our other CRs. In such cases let us know the use case and which feature you are missing in our current CRs.
  • The native Envoy HTTP filter and cluster are defined in the value section in raw types. The Microgateway Operator validates only the YAML syntax and if the input can be unmarshalled to the corresponding proto type.
 
Info

Consult the Envoy documentation Envoy clusters for further information.

 
Functional limitation

By default, every CustomResource SidecarGateway results in a cluster named backend in the Envoy configuration, which points to the upstream application.

  • Clusters with the same name overwrite previously defined clusters!
  • It is not possible to overwrite the Airlock Microgateway default backend cluster.

CR EnvoyConfiguration

The CR EnvoyConfiguration is the schema for the Envoy configurations API.

 
Notice

This CRD is used for internal purposes. EnvoyConfiguration resources are not intended to be manually created, modified, or deleted by the user.

 
Risk

EnvoyConfiguration resources may contain sensitive information, and thus RBAC permissions should be granted with care.

CR EnvoyHTTPFilter

The CR EnvoyHTTPFilter is an additional Envoy HTTP Filter resource that is added to those defined by the Airlock Microgateway. If a feature is available as a native Envoy HTTP filter but is missing in the Microgateway CRs, the CR EnvoyHTTPFilter can be used to prepend the native Envoy HTTP filter. For the CRs EnvoyHTTPFilter and EnvoyCluster, the Envoy configuration language must be used.
For an example targeting debugging, see article Debugging with Envoy filters.

Applicable data plane mode

  • Sidecar-based data plane mode.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

References

Prerequisites

-

 
Notice
  • You should use the CRs EnvoyHTTPFilter and EnvoyClusters only if a feature is unavailable in our other CRs. In such cases let us know the use case and which feature you are missing in our current CRs.
  • The native Envoy HTTP filter and cluster are defined in the value section in raw types. The Microgateway Operator validates only the YAML syntax and if the input can be unmarshalled to the corresponding proto type.
 
Info

Consult the Envoy documentation Official Envoy HTTP filters documentation for further information.

CR GatewayParameters

The CR GatewayParameters defines the configuration settings for deploying a Gateway in Kubernetes, including options for logging, service type, deployment strategy, and resource management.

Applicable data plane mode

  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

GatewayClass.spec.parametersRef.name

Example:

 
Example
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: airlock–microgateway
  namespace: airlock–gateway 
spec:
  controllerName: microgateway.airlock.com/gatewayclass–controller
  parametersRef:
    group: microgateway.airlock.com
    kind: GatewayParameters
    name: airlock–microgateway 
    namespace: airlock–gateway

Required incoming reference from GatewayClass.

Gateway.spec.infrastructure.parametersRef.name

Example:

 
Example
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway–secure
  namespace: airlock–gateway
spec:
  gatewayClassName: airlock–microgateway
  listeners:
  – name: http–listener
    protocol: HTTP
    port: 80
  infrastructure:
    parametersRef:
      group: microgateway.airlock.com
      kind: GatewayParameters
      name: airlock–microgateway 
      namespace: airlock–gateway  

Required incoming reference from Gateway.

References

This CR contains the configuration for session handling.

Prerequisites

-

 
Notice

More specific CR GatewayParameters take precedence, e.g., if a Gateway resource references GatewayParameters, the parameters from its associated GatewayClass are completely overridden without merging.

CR GraphQL

The CR GraphQL allows referencing a GraphQL schema resource. The schema of this ConfigMap resource must be compatible with GraphQL schema definition (Oct. 2021). Introspection and mutation queries are supported.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference.

References

Prerequisites

-

GraphQL queries, variables and operation names can be extracted from different sources:

  • HTTP query parameters
  • JSON bodies

Not all configurational aspects of GraphQL–related traffic are configured in the CR GraphQL. Even without a referenced schema, CR DenyRules and CR Limits settings are applied to query requests and can be configured to block or restrict them.

GraphQL queries are identified with parameters and keys and must match the parser's requirements – see CR Parser.

In case of a problem, such as an error in the referenced GraphQL schema, Airlock Microgateway logs information in the application log.

Example:

 
Example
[2024–06–12 14:08:43.436][125742][critical][main] [external/envoy/source/server/server.cc:414] error initializing config '  /home/mgw/airlock/airlock–microgateway–engine/config/graphql.yaml': Failed to parse GraphQL configuration "graphql_config": schema parse error: Parse error at 1:16 
Unexpected `abc[Name]` 
Expected `:`

CR HeaderRewrites

The CR HeaderRewrites is the Schema for the header rewrites API.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

Prerequisites

-

  • The predefined built–in header lists are constantly evolving and updated to respond to the latest threats and new requirements.
  • Custom header rewrite rules can be added to address various integration requirements and to prevent information leakage by response headers and attacks by request headers.

With the configuration options, request and response headers can be added to or removed from upstream and downstream traffic.

The content of this CR is split into two main sections request and response. Both sections can be complemented with custom rules.

Request/response headers are processed in the following order:

  1. Remove all headers, that are not explicitly listed in a built–in or a custom allow rule.
  2. Remove all headers, that are explicitly listed in a built–in or a custom remove rule.
  3. Add all headers, that are listed in a built–in or custom add rule.

The following actions can be configured for both built–in and custom rules:

Configured action

Additional information

allow

  • It is possible to disable filtering based on header allow lists using the allHeaders: {} option. In this case, you should use an equivalent set of remove rules to avoid unwanted information leakage through response headers or attacks through request headers.
  • Filtering with header allow lists can be enabled using the matchingHeaders configuration element.

remove

  • The configuration options for rules in the remove section are similar to those in the allow section.

add

  • To add a header with the given name and value. In addition, a mode parameter can be added to define the behavior in case the header already exists.
  • It is set to ​AddIfAbsent​ by default but could be set to OverwriteOrAdd.

Logging
Header rewrite actions are not displayed in the access log by default. The CR HeaderRewrites features an Integration operation mode that enriches the access log output with information and details not logged in Production mode. The additional log information are useful for application integration and fault–finding tasks, i.e., to discover potential header–related issues.

Note that the extended logging output of the Integration mode results in larger log messages and requires more storage.

Example log output in Integration mode:

 
Example
   "airlock": { 
    "header_rewrites": { 
      "response": [ 
        { 
          "headers": [ 
            "content–type" 
          ], 
          "type": "remove–rule", 
          "applied_rules": [ 
            "Remove Content Type Response Header" 
          ], 
          "action": "remove" 
        } 
      ], 
      "request": [ 
        { 
          "type": "allow–rule", 
          "applied_rules": [ 
            "Standard Allowed Request Headers", 
            "Allow custom request headers", 
            "Propagate X–Forwarded–For to upstream" 
          ], 
          "action": "remove", 
          "headers": [ 
            "front–end–https", 
            "x–forwarded–host", 
            "x–forwarded–port", 
            "x–forwarded–proto", 
            "x–forwarded–server", 
            "x–real–ip", 
            "accept–encoding" 
          ] 
        } 
      ] 
    },

CR IdentityPropagation

The CR IdentityPropagation specifies how the identity of the authenticated user is propagated from the Microgateway Engine to the back–end. For example, it is possible to set a header containing an OIDC idToken claim.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

Prerequisites

-

 
Info

For advanced use cases propagation values can also be extracted from Envoy dynamic metadata.

CR JWKS

The CR JWKS provides the configuration for JSON Web Key Set providers and endpoints, including certificate verification settings.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference to verify the JWT.

Required incoming reference to verify the id token in the OIDC flow.

References

Prerequisites

-

CR JWT

The CR JWT configures client authentication via a JSON Web Token.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

This CR contains the configuration to verify the JWT.

Prerequisites

-

CR Limits

Airlock Microgateway performs various size checks on requests with a default configuration to repel denial of service (DoS) attacks. The CR Limits can be used to configure these limit checks on the root level. All limits are calculated in bytes.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

Prerequisites

-

Limit checks are available for:

  • Number and size of parameters.
  • Request properties like body size or path length.
  • Various aspects of JSON Payload.
  • A global header size check (can be configured in the CR SidecarGateway).
  • Configured limits can also apply to OpenAPI and GraphQL requests.

  • WebSocket and gRPC currently use an unlimited (hard–coded) body size.
  • The pathLength limit is a general type and applies for parsed and unparsed requests.
  • JSON limits will only be applied if the JSON parser in the CR Parser is enabled.
  • Body size limits are applied as general.bodySize for unparsed requests (e.g., requests with unknown content type) or as individual bodySize​ per feature (e.g., multipart or JSON bodies).
  • Other limits, such as parameter size, are counted in parsed form.

Size limits like bodySize, nameLength , etc., can be specified in bytes or Kubernetes memory limit units.

Examples of Kubernetes memory limits:

  • 1k = 1'000 bytes
  • 1Ki = 1'024 bytes
  • 1M = 1'000'000 bytes
  • 1Mi = 1'048'576 bytes
 
Risk

Limits can be deactivated with unlimited {} if not hard–coded. Keep in mind that this disables denial of service (DoS) protection and, therefore, is not recommended in production environments.

Threat handling mode settings
When a request causes a block due to reaching the request size limit in Block mode, it is possible that the total request size is not fully logged. For example, it can happen that the actual parameter name length is not part of the logging information because the request size limit has already been reached before and the request is already blocked. This behavior is to mitigate the risk of being vulnerable to DoS attacks.

 
Risk

Detailed limit violations can be logged with settings.threatHandlingMode: LogOnly. Keep in mind that this disables the denial of service (DoS) protection and therefore is not recommended in production environments.

The logs may vary depending on the settings.threatHandlingMode. In Block mode, only the available information is logged to still protect against denial of service attacks. This could lead to some information being missing whereas in LogOnly mode, all limits–related information is logged.

The following examples show two types of limit violations, body size and JSON key length, in both threat–handling modes.
As expected, the body.size value is not shown in Block mode in opposition to the JSON key.length value (which requires parsing the JSON body for length calculation).

Block mode

LogOnly mode

 
Example
# body size, block 
 
    "limits": { 
      "matches": [ 
        { 
          "blocked_data": { 
            "body": {} 
          }, 
          "rule": "Built–in: Request body size", 
          "threat_handling_mode": "block" 
        } 
      ] 
    },
 
Example
# body size, logOnly 
 
    "limits": { 
      "matches": [ 
        { 
          "blocked_data": { 
            "body": { 
              "size": 219 
            } 
          }, 
          "rule": "Built–in: Request body size", 
          "threat_handling_mode": "logOnly" 
        } 
      ] 
    },
 
Example
# JSON key length, block 
 
    "limits": { 
      "matches": [ 
        { 
          "blocked_data": { 
            "json_key": { 
              "json_path": "$['username']", 
              "key": "username", 
              "length": 12 
            } 
          }, 
          "rule": "Built–in: JSON key length", 
          "threat_handling_mode": "block" 
        } 
      ] 
    },
 
Example
# JSON key length, logOnly 
 
    "limits": { 
      "matches": [ 
        { 
          "blocked_data": { 
            "json_key": { 
              "json_path": "$['username']", 
              "key": "username", 
              "length": 12 
            } 
          }, 
          "rule": "Built–in: JSON key length", 
          "threat_handling_mode": "logOnly" 
        } 
      ] 
    }, 

The article Access log field reference provides additional information listing all log fields including short descriptions.

 
Functional limitation

If the request's body is not in UTF8, the body–size calculation will fail due to an internal check. In this case, no information about the body size is available.

Limit–based block and attack types in logs and metrics
In addition to the obvious block type limits, limit–based blocks (or potential blocks in LogOnly mode) can occur based on other Microgateway features that also make use of the configured limits and appear in logs and metrics. All configured limit types can trigger a single request block or multiple potential blocks in LogOnly mode.

At the time of writing, limit–related block or log counts can also appear based on the block_type labels:

  • openapi
  • graphql

For example, Airlock Microgateway Engine metrics can contain the following metric labels:

 
Example
microgateway_http_downstream_rq_threats_blocked_total{block_type="graphql", attack_type="parameter_body_size", envoy_cluster_name="YourClusterName"}

The metrics block count in the example above has been triggered by a graphql query in a JSON body exceeding the parameter_body_size limit. The label metrics naming is closely related to the CR Limits configuration for intuitive understanding. For example, if a request exceeds the request.limited.general.bodySize value, the corresponding metric label for attack types would be general_body_size.

CR OIDCProvider

The CR ODICProvider specifies an OpenID Connect Provider (OP), such as Airlock IAM, with the required authorization and token validation endpoints.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

This CR contains the configuration to verify the id token in the OIDC flow.

Prerequisites

This CR contains the configuration for session handling and is required to store relevant information about the authentication.

About the example configuration:

 
Notice

When using self–signed TLS certificates for securing the OP endpoints, tls settings must be configured accordingly.

CR OIDCRelyingParty

The CR OIDCRelyingParty specifies how the Airlock Microgateway Engine interacts with an OpenID Provider (OP) such as Airlock IAM.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

This CR contains the configuration for the OIDC Provider (OP).

Prerequisites

This CR contains the configuration for session handling and is required to store relevant information about the authentication.

About the example configuration:

  • The referenced CR OIDCProvider specifies the OpenID Connect provider, e.g., Airlock IAM. Currently, only client secret–based authentication via ​BasicAuth or ​FormURLEncoded POST body is available.
  • When accessing the logoutPath, the user is logged out from the OIDC relying party. If required, logout from the OIDC provider must be triggered separately.

The Secret requires the key client.secret for the password string as in the following example:

 
Example
apiVersion: v1 
kind: Secret 
metadata: 
  name: oidc–client–secret 
stringData: 
  client.secret: "myClientSecret1234"

CR OpenAPI

The CR OpenAPI contains the configuration for the OpenAPI specification.
Enforcing a tight OpenAPI specification reduces the attack surface significantly. HTTP requests and responses are checked against the OpenAPI specification and will be blocked in case of a violation.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference.

References

Prerequisites

-

 
Info

OpenAPI specifications are typically generated in an automated way by the API build pipeline. No reverse–engineering by security personnel is required and a DevSecOps process can be established.

 
Info

There are two main error cases with missing or faulty OpenAPI specifications:

  • If a specification cannot be parsed, an application log message is written, e.g., Failed to parse openapi config. JSON parse error: Missing a comma or '}'....
  • If an empty OpenAPI specification is configured, the access log message OpenAPI schema is empty is written and all requests are rejected.

Logging
In case of a problem in the configured OpenAPI specification, Airlock Microgateway logs information in the application log.

  • JSON syntax error example in OpenAPI specification:
  •  
    Example
    [2023–09–25 11:58:25.476][37802][warning][config] [airlock/extensions/filters/http/openapi/openapi_filter_config.cc:125] Failed to parse OpenAPI configuration "<OpenAPI–reference–name>": JSON parse error: Missing a name for object member. offset 36
  • OpenAPI syntax error example:
  •  
    Example
    [2023–09–25 11:59:49.918][38151][critical][main] [external/envoy/source/server/server.cc:134] error initializing config '  /home/mgw/airlock–microgateway–engine/config/openapi.yaml': Failed to parse OpenAPI configuration "<OpenAPI–reference–name>": Expected key "in" not found $.paths./WebApplication.post.parameters[0]

Feature scope and configuration options

The OpenAPI filter supports the validation of requests and responses:

  • Path
  • Query parameters
  • Headers
  • Cookies
  • JSON syntax and objects
  • Body content checks are only applied to JSON documents, form–urlencoded parameters, and binary data (jpgs, gifs, ...).

  • API specification format must be in OpenAPI version 3.0 as JSON.

 
Functional limitation

The OpenAPI feature does currently not support the following:

  • For requests with form–urlencoded parameters, additionalProperties can only be configured as a boolean.
  • XML and YAML content types are unsupported. These content types are passed unchecked, even when specified in a schema.
  • Multipart requests.
  • Callback definitions.
  • OpenAPI schema in YAML format.
 
Info

Specifications in other formats or versions, e.g., Swagger 2.0, must be converted before uploading. For Swagger to OpenAPI conversions, we recommend the Mermade converter, which is available as a command-line tool.

CR Parser

The CR Parser contains the configuration for content parsers (default and custom).
Airlock Microgateway Engine parses the request body before applying filters (deny rules, limits, OpenAPI, GraphQL) to a request's content. The parser selection is based on the Content–Type header.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

Prerequisites

-

  • Built–in parsers can be enabled/disabled individually (e.g., JSON and URL–encoded form parsers).
  • The built–in logic parses requests with Content–Type headers matching the corresponding mediaTypePattern pattern. The default pattern can be adjusted to trigger parsing on other content–type patterns.
  • Multipart parsing is enabled by default in the CR Parser. The Engines deny rules are applied to the parsed parameters as configured.
  • For requests without a Content–Type header, the parser logic adds a content–type header with the value configured in defaultContentType. However, the default can be customized using the option. The added Content–Type header is also used to select the correct parser as described above.
 
Risk

The parser configuration is a crucial setting and influences the behavior of other filters. For example, disabling the JSON parser or set a less restrictive value causes that deny rules, limits, OpenAPI specification, or GraphQL schema validation are not applied at all. The same is true for other parsers.

GraphQL query parsing
The parser can identify and parse GraphQL queries from different types of requests.

  • JSON bodies (with content type application/json). Note that the JSON parser must be enabled.
  • HTTP requests with query parameters

The parameter names/JSON keys are “query”, “variables” and “operationName”, where the former is required for GraphQL query identification.

The key “operationName” is used as an identifier for logging if present.

CR RedisProvider

The CR RedisProvider contains a client configuration for connecting to a Redis database. For caching session handling information, a Redis database can be deployed within or outside the Microgateway cluster.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference.

References

Prerequisites

-

 
Risk

We highly recommend enabling SSL/TLS encrypted communication between the Airlock Microgateway and the Redis database, especially when using an external database, to keep the session information secure. Note that both components, the Airlock Microgateway and the Redis database, must have the same TLS configuration for communication to be possible.

With tls: {} enabled only certificates which are signed by a CA that is trusted by any of the root CA certificates, built into the Microgateway Session Agent’s base image, are accepted until configured otherwise.

CR SessionHandling

The CR SessionHandling contains the configuration for session handling and specifies the Redis database where sessions should be stored.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

This CR contains the client configuration for connecting to a Redis database.

Prerequisites

A Redis database must be configured for session handling.

 
Info

Session sharing between different Microgateway deployments, i.e., for single sign-on (SSO), is possible. It can be achieved by configuring the prefix and persistence back–end setting to be the same across all corresponding SessionHandling CRs or using a shared SessionHandling CR.

 
Functional limitation

Session cookies are currently not shared across different FQDNs (fully qualified domain names). Authentication via different OIDC Relying Parties generates different session cookies.

Clients can only transparently reuse session cookies for connecting to different Airlock Microgateway deployments if the cookies are:

  1. exposed under the same FQDN
  2. and

  3. handle authentication via the same OIDC Relying Party.

CR SidecarGateway

The CR SidecarGateway contains the configuration of how to configure the Airlock Microgateway Engine when used as a Sidecar Container within the Pod of an application.

Applicable data plane mode

  • Sidecar-based data plane mode.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

References

This CR contains the configuration for authentication, authorization and identity propagation.

This CR specifies the options to secure an upstream web application.

This CR contains the configuration for additional Envoy HTTP Filters.

This CR contains the configuration for additional Envoy HTTP Clusters.

This CR contains the configuration for the telemetry (logging, metrics and tracing).

This CR contains the configuration for session handling.

Prerequisites

-

 
Notice

Without a Custom Resource SidecarGateway referring to a Pod, the Microgateway Engine does not forward any traffic to the upstream application. This results in Connection Refused errors on access.

 
Notice

The status of referenced CRs (including nested CRs) is available via CR SidecarGateway status – missing references are reported with further information in the form of a message and a reason. This helps to find problems and solve them quickly.

The following example shows a simple CR SidecarGateway configuration.

It is important to understand how Airlock Microgateway Engine configuration and security settings are applied to web applications:

  • When a web application is deployed with an inject label (see example below), a Microgateway Engine sidecar is injected by the Microgateway Operator during Pod creation.
  • To apply Microgateway Engine settings to a web application Pod, a corresponding CR SidecarGateway is required.
  • When multiple web application pods with Microgateway Engine sidecars are deployed in the same namespace, multiple SidecarGateway CRs can be created.
  • Labels of the web application Pod can be added to the corresponding CR SidecarGateway under spec.podSelector.matchLabels. Reverse lookup determines the pod(s) whose labels match the matchLabels in the CR.

Example Deployment:

 
Example
apiVersion: apps/v1  
kind: Deployment  
metadata:  
  labels:  
    app: echo–server  
  name: echo–server  
spec:  
  selector:  
    matchLabels:  
      app: echo–server  
  template:  
    metadata:  
      annotations:  
        sidecar.microgateway.airlock.com/excludeInboundPorts: "9998,9999" 
        sidecar.microgateway.airlock.com/serviceMesh: "none" 
      labels:  
        sidecar.microgateway.airlock.com/inject: "true"  
        app: echo–server  
        version: 1.0.0
    spec:
      ...    

Example SidecarGateway:

 
Example
apiVersion: microgateway.airlock.com/v1alpha1 
kind: SidecarGateway 
metadata: 
  name: sidecar–gateway–example 
spec: 
  podSelector: 
    matchLabels: 
      app: example–app 
  applications: 
    – containerPort: 5051
      routes: 
        – pathPrefix: / 
          secured: 
            contentSecurityRef: 
              name: content–security–example   

SidecarGateway configuration to protect a web application:

  • Our example uses a simple web application Pod with the label app: example–app.
  • The spec.applications.containerPort is the container port of the web application that has to be secured by Airlock Microgateway. The default 8080 can be overridden to connect to the application's HTTP port. In our example, the example–app Pod listens for requests on port 5051.
  • The security settings to be applied are referenced in the CR ContentSecurity named content–security–example.
 
Notice

TLS is disabled by default for both downstream and upstream traffic.

In the downstream section (receiving requests):

  • With TLS enabled, set the required secretRef to a TLS Secret that contains a server certificate.
  • Optionally, configure the TLS protocol version. However, the defaults should be fine for most cases and can be looked up at Envoy TLS minimum/maximum protocol settings.
  • The TLS ciphers can be configured, if required. However, the default ciphers should be fine for most cases. Additional information on envoy cipher suite configuration can be found at Envoy cipher suites (BoringSSL-based).

For more information, see also TLS secrets in Kubernetes.

In the upstream section (connections to the application):

  • Enabling TLS for upstream establishes TLS connections between the Microgateway Engine and the upstream application.
  • Optionally, configure the TLS protocol versions. However, the defaults should be fine for most cases and can be looked up at Envoy TLS minimum/maximum protocol settings.
  • The TLS ciphers can be configured, if required. However, the default ciphers should be fine for most cases. Additional information on envoy cipher suite configuration can be found at Envoy cipher suites (BoringSSL-based).

For the default and a simple example configuration, see CR SidecarGateway reference documentation, spec.applications.downstream.tls or spec.applications.upstream.tls.

Client certificate settings for downstream mTLS
To secure access to the application with mutual TLS (mTLS), Airlock Microgateway needs to instruct clients to provide a valid client certificate during a TLS handshake. The exact behavior can be configured using the downstream.tls.clientCertificate option.

Client certificate handling options

Description

ignored: {}

  • Disables the verification of client certificates (default).

optional:

  • Client certificates will be verified if presented by the client.
  • A client certificate is not required to access the application.

required:

  • Clients must present a valid client certificate.
  • Additional settings for client certificate verification can be configured.

The following example shows available client certificate verification settings:

 
Example
...
      downstream:
        tls:
          enable: true
          secretRef:
            name: server–cert–secret
          clientCertificate:
            required: 
              trustedCA:
                verificationDepth: 1
                certificates:
                  – secretRef:
                      name:  trusted–ca–secret 
              crl:
                validationMode: VerifyChain
                lists:                                   
                  – secretRef:
                      name: crl–secret  
              allowedSANs:
                – sanType: DNS
                  matcher: 
                    exact: "api.example.com"
                                            
              certificatePinning: 
                allowedHashes: 
                – a434460ca545...
                allowedSPKIs: 
                – seFTOlIPG28LywSNn6wbPqRiV6GRJ66z+0J0Cwt2lMQ=
... 

Use the following table as a starting point when configuring the Secret files required for mTLS.

Resource:

Description:

Example:

trustedCA

  • List of secrets containing trusted CA certificates.

The certificate is expected in a generic secret with the key ca.crt:

 
Example
apiVersion: v1
kind: Secret
metadata:
  name: trusted–ca–secret
data:
  ca.crt: <base64 encoded ca certificate>

crl

  • List of secrets containing a Certificate Revocation List (CRL) in PEM format.

The CRLs are expected in a generic secret with the key ca.crl:

 
Example
apiVersion: v1
kind: Secret
metadata:
  name: crl–secret
data:
  ca.crl: <base64 encoded crl>

For the default and example configuration, see CR SidecarGateway reference documentation, spec.applications.downstream.tls.

HTTP protocol settings:
The HTTP protocol version used for downstream and upstream traffic is automatically selected by default based on the capabilities of upstream and downstream systems.

 
Info

If the default HTTP auto selection does not work as expected, try an explicit protocol version configuration instead. See article HTTP protocol selection.

In the downstream section (connections from clients):

  • The default ​auto setting allows requests with all HTTP protocol versions.
  • Protocol versions can be configured explicitly. Note that with an explicit protocol version, non–matching requests may be blocked.

In the upstream section (connections to application):

  • With the default auto setting, the Microgateway Engine selects the most appropriate HTTP protocol version.
  • Protocol versions can be configured explicitly. Note that the application may block requests with an explicit protocol version due to a version mismatch.

For the default and example configuration, see CR SidecarGateway reference documentation, spec.applications.downstream.protocol or spec.applications.upstream.protocol.

Downstream remote IP settings:
In Kubernetes/Openshift environments, it is extremely unlikely for an external client to have a direct connection to an application. Usually, requests pass through one or more reverse proxy layers, hiding the client's real IP address. Sometimes, the client remote IP is an input parameter to Microgateway features like deny rule exceptions based on a client IP range. Other times, it is simply important to log the correct client IP.

Therefore, the remoteIP settings define how the remote IP address of a client is identified.

The remote IP extraction may be configured with either of these options:

  1. With connectionIP: {} the remoteIP address of the directly connected client will be used.
  2. The alternative customHeader option can only hold a single client IP address (IPv4/IPv6).
    If required is set to true, the Microgateway Engine will block the request if the custom header is missing.
  3. The default and most common option is to have upfront reverse proxies include the client IP address in an xff (X–Forwarded–For) header. The standard actually allows the header to hold multiple IPv4/IPv6 addresses.
    If a request passes through a proxy, it may add its IP address to the right side of the address list in the xff header.
    The numTrustedHops defines which item in the list will be selected as the remote IP address. It counts from right and defaults to 1, which selects the rightmost entry in the IP list in the header.
    For example, 203.0.113.59 would be selected by default for the header X–Forwarded–For: 203.0.113.182,150.172.238.178,203.0.113.59.
  4. Note that the connectionIP will be used as a fallback in case the XFF header contains fewer IPs than configured as numTrustedHops.

For the default and example configuration, see CR SidecarGateway reference documentation, spec.applications.downstream.

Request normalization settings:
With the option requestNormalizations it is possible to transform the request path before it is matched or routed. See Envoy path transformation for more information.

For the default and example configuration, see CR SidecarGateway reference documentation, spec.applications.downstream.

Limiting request headers length:
The maximum size of the request header block, encompassing all headers and their values, can be restricted by setting a header length limit. Additional limits on requests can be configured in the CR Limits.

For the default and example configuration, see CR SidecarGateway reference documentation, spec.applications.downstream.restrictions.http.

Filtering:
By default, all traffic is filtered and secured using reasonable default security settings. However, with todays web applications consisting of different parts, for example HTML pages and REST APIs, with different requirements, specific security policies can be applied using CR ContentSecurity.

  • Microgateway Engine filtering can even be bypassed completely by explicitly setting a route to unsecured{}.
  • Since the first matching route will be applied to every request, the order of route configuration entries is significant, i.e., a fallback path has to be placed at the end of the routes list.
 
Risk

If you exclude a route by setting it unsecured{}, the target application is exposed to untrusted downstream traffic. Use this option with caution!

CR Telemetry

The CR Telemetry contains the configuration for telemetry (logging, metrics, and tracing).

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

References

Prerequisites

-

The CR Telemetry can be used to customize the default log format of Airlock Microgateway.

Use cases for a custom log format:

  • To reduce the log volume.
  • To include additional log information such as request cookies, request headers, ...
  • To reorganize the format to simplify the correlation with logs of other infrastructure components.

The CR Telemetry reference documentation contains the default access log configuration and an example configuration. The default configuration can be used as a starting point for customization.

The access log format can be configured in the Envoy format dictionary style. Both command operators, standard Envoy command operators and Airlock Microgateway custom command operators, may be used in the configuration.

Log correlation:
The behavior of Airlock Microgateway Engine when receiving an x–request–id header can be configured with the keys allowDownstreamRequestID and alterRequestID under the setting correlation.request. These two settings allow configuring the booleans explained in Envoy UUID request id configuration.

 
Notice

Remember that documented examples, dashboards, explanations, or alike always refer to our default log format.

CR TokenExchange

The CR TokenExchange configures an OAuth Token Exchange (RFC 8693). This feature may be used when the upstream backend requires a different token than the downstream JWT or OIDC Access Token.

Applicable data plane mode

  • Sidecar-based data plane mode.
  • Sidecarless data plane mode with K8s Gateway API.

API reference

Consider the API reference for the defaults, all options, and a configuration example.

Referenced by

Required incoming reference in sidecar-based data plane mode.

Required incoming reference in sidecarless data plane mode.

References

Prerequisites

-

 
Notice

Token Exchange is used to exchange the received token from the downstream to a new one which could be propagated to the upstream service.

The token from the downstream are received from one of the following sources:

Important: The token must be expliclity propagated using CR IdentityPropagation. Otherwise, the exchanged token will only remain in Microgateway's memory.

 
Info

The Airlock Microgateway implements token caching to reduce the load on the Token Exchange Server. While this caching behavior is not directly configurable, it is implicitly governed by session management settings.

Caching conditions

  • Token caching is activated only when session handling is enabled. This is managed through the CR SessionHandling configuration.
  • Once the token has been exchanged, this token is cached in the session handling persistence. As a result, the duration for which the token remains cached is influenced by the following session parameters:
    • Lifetime: Configurable via the CR SessionHandling setting.
    • Idle timeout: Fixed at 600 seconds and cannot be changed.

These settings determine how long the cached token is retained and whether a fresh token exchange will be triggered.

Logging cached tokens

Each request logs whether a token was reused from the cache or newly exchanged. This information can be found in the access log entry under the path: airlock.access_control.token_exchange.sequence[].cached

If “cached”: true, it indicates that the token was reused from the session cache.

Example access log entry

{
  "airlock": {
    "access_control": {
      "policy": "policy_name",
      "authenticated": true,
      "authorized": true,
      "type": "jwt",
      "user_id": "...",
      "status": "complete",
      "details": {
        "jwt": {
          "token": { ... },
          ...
        },
        "token_exchange": {
          "sequence": [
            { "type": "jwt", "iss": "...", "sub": "...", "cached": true }
          ]
        }
      }
    }
  }
}