Troubleshooting for network routing issues in the application pod

The articles in this chapter cover Microgateway CNI DaemonSet, Microgateway CNI plugin and the Microgateway Network Validator issues in a sidecar-based installation and help with troubleshooting.

Routing configuration schematics

Routing all traffic through the Microgateway Engine is crucial for application security. The following drawing is a simplified schematic of the components working together for routing configuration and the initial routing sanity check.

CNI installation and verification
  1. The Microgateway CNI DaemonSet installs the Microgateway CNI plugin on each node (except the nodeAffinity or nodeSelector settings are configured otherwise).
  2. Labelled web applications are deployed with a Microgateway Engine and a Microgateway Network Validator. The Microgateway CNI plugin configures the routing within these web application Pods.
  3. The Microgateway Network Validator initContainer performs a sanity check that fails if the Microgateway CNI plugin is not working/installed or is not ready when the web application pod is deployed and running.
  4. Depending on the config.repairMode configured for the Airlock Microgateway CNI Helm chart, the CNI Installer will try to repair Pods with failing Network Validators.

Network routing can fail for different reasons, mainly configuration issues. A good starting point is to check the basic configuration requirements as listed below.

Basic requirements

Check the following basic configuration requirements, depending on your deployment.

  • Web application Pod:
  • For application Pods hostNetwork: false must be configured. Otherwise, the Microgateway CNI plugin is not executed.
    The hostNetwork: true configuration option is currently unsupported by Airlock Microgateway.
  • Install the web application pod only on nodes that have the Microgateway CNI plugin installed. Check the nodeAffinity and nodeSelector of the Airlock Microgateway CNI and your application.
  • Microgateway Network Validator:
  • Port 19003 must be reserved in the web application Pod for the Network Validator initContainer. The port is required for the initial sanity check of the routing in the web application Pod.
  • Microgateway CNI plugin:
  • The Sidecar-based preparations must be installed on the node and be in working order before the web application Pod is deployed.
  • Optional: A nodeAffinity and/or a nodeSelector can be configured in the CNI Helm chart so that the Microgateway CNI plugin is only installed on nodes with matching labels. See official Kubernetes node affinity documentation for more information.

Recommended quick check sequence

The following list is logically ordered for fast fault finding if you encounter Microgateway CNI or routing problems.

  1. As part of each Airlock Microgateway installation, ensure the Microgateway CNI DaemonSet is properly installed and running.
    • Before installation:
    • Ensure the chosen <environment>-values.yaml preset (GKE, OpenShift, ..) is correct for your cluster type.
    • Ensure the version of the Microgateway CNI DaemonSet and the Microgateway Operator match to avoid incompatibilities (e.g., with updated routing configuration).
  2. Ensure the Microgateway CNI plugin is installed on the target node when the web application with the Microgateway Engine is deployed, as described section Sidecar-based preparations.
  3. Check the state of the Microgateway Network Validator container inside the web application Pod.

Network Validator sanity check

The Microgateway Network Validator is an initContainer deployed in the web application container. It uses the Microgateway Engine image with a different entrypoint. During the startup of the web application container, the initContainer generates a test request to verify that the Microgateway CNI plugin has configured the routing inside the application container.

  1. Status verification of the Microgateway Network Validator:
  2. On the application namespace, run kubectl describe pod ....
    Example output of a failed sanity check:
  3. copy
    ...
    Init Containers:
      airlock-microgateway-network-validator:
        Container ID:    containerd://47190516e4378b982f3ee854b60c3c95d933c1882b83bda246fa0352006e453e
        Image:           quay.io/airlock/microgateway-engine@sha256:6051975a14c51b9d3b525a06004d62a4d323c08ca58e3468343095a55a42fff2
        ...
        State:          Waiting
          Reason:       CrashLoopBackOff
        Last State:     Terminated
          Reason:       Error
          Message:      Traffic redirection to Airlock Microgateway Engine is not working.
    Restart the pod after ensuring that hostNetwork is disabled and a compatible Airlock Microgateway CNI version is installed on the node.
    Certain environments may also require additional configuration (see docs.airlock.com for more information).
          Exit Code:    1
          Started:      Fri, 03 Nov 2023 12:23:13 +0100
          Finished:     Fri, 03 Nov 2023 12:23:13 +0100
        Ready:          False 
    ...
  4. Check for the status (here: terminated with error) and the message text of the airlock-microgateway-network-validator container for more information. Note that the message depends on the problem detected and can differ from the example above.
  5. If the web application Pod status fails due to a terminated Network Validator, this is a strong hint for a CNI plugin problem. We recommend checking the logs and installing the Microgateway CNI plugin before the web application Pod is deployed and running.

Testing a deployed web application

For more extensive testing of the deployment, routing and functionality of the Microgateway Engine, we recommend using test-requests to cover all three mentioned aspects.

  1. Secure a web application with Airlock Microgateway.
  2. Check the filtering and the traffic routing through the Microgateway Engine:
    • Send one or more requests to the web application.
    • Check the Microgateway logs to see whether the request has been logged.
    copy
    kubectl logs <POD> -c airlock-microgateway-engine
    

    Example output:

    details...
    {
      "@timestamp": "2025-01-14T15:23:10.117+0000",
      "ecs": {
        "version": "8.5"
      },
      "observer": {
        "version": "4.5.0",
        "product": "Airlock Microgateway",
        "type": "waap",
        "vendor": "Ergon Informatik AG"
      },
      "log": {
        "logger": "access",
        "level": "info"
      },
      "http": {
        "version": "1.1",
        "response": {
          "bytes": 659,
          "body": {
            "bytes": 53
          },
          "mime_type": "text/plain",
          "status_code": 400
        },
        "request": {
          "method": "GET",
          "bytes": 151,
          "id": "d7fd95e2-c506-96e6-94b8-49aa40092d24",
          "body": {
            "bytes": 0
          }
        }
      },
      "airlock": {
        "summary": {
          "details": "airlock_request_blocked_deny_rule",
          "flags": "-",
          "action": "denied"
        },
        "log_correlation": {
          "connection_id": 519,
          "stream_id": "8288591618986530680"
        },
        "actions": {
          "block": {
            "block_type": "deny_rules",
            "block_subtype": "XSS",
            "details": {
              "deny_rules": {
                "level": "strict",
                "rule_key": "XSS",
                "blocked_data": {
                  "path": {
                    "path": "/\u003cscript\u003e"
                  }
                },
                "rule_id": 5
              }
            }
          }
        }
      }, 
      ...
      ...
    }