Analyze Configuration Issues

General approach

Troubleshooting Microgateway resources follows the same problem-solving approach as troubleshooting other Kubernetes workloads:

1. Check resource conditions for potential problems.

   1. Start with inspecting the Gateway resource.
      Note: If the Gateway resource is not accessible due to missing permissions, skip this step.
   2. Continue with inspecting HTTPRoute resources.
   3. Finish with inspecting policy resources.

2. Review events that may point to pending or unresolved issues.

3. Inspect logs of the involved components for traces or details of faulty behavior.

While kubectl can be used for all of these tasks, this guide uses gwctl, a tool specifically designed for Kubernetes Gateway API resources.

Check Gateway resource conditions

Getting a status overview

1. List a gateway together with its Programmed condition, the number of attached routes, and the directly attached policies:

   1 2 3

   gwctl get gateway -n example-ns -o wide NAMESPACE NAME CLASS ADDRESSES PORTS PROGRAMMED AGE POLICIES HTTPROUTES example-ns mygateway airlock-microgateway 100.126.1.2 80,443,8443 True 4h28m 0 3

   A Programmed condition set to False, or an unexpected number of listed policies or accepted HTTPRoutes, indicates that something is wrong.

2. Use gwctl describe for detailed information on the status of the gateway, its listeners, the attached routes, and their back-ends.

Verifying the GatewayClass

1. Make sure that the gateway class exists and is correctly referenced by its name.

2. Use gwctl get gatewayclasses to list the available gateway classes:

   1 2 3

   gwctl get gatewayclasses NAME CONTROLLER ACCEPTED AGE airlock-microgateway microgateway.airlock.com/gatewayclass-controller True 5h21m

   The value in the NAME column must match the CLASS column in the gwctl get gateway output.

Verifying the license validity

If you use the Community Edition, you can skip the license validity check.

1. Verify that the gateway exposes a dedicated condition that reports the status of the configured Microgateway license.

2. Confirm that the license is valid:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

   gwctl describe gateway -n example-ns mygateway Name: mygateway Namespace: example-ns ... Status: addresses: - type: IPAddress value: 100.126.1.2 conditions: ... - lastTransitionTime: "2025-11-26T15:23:06Z" message: 'Valid Airlock Microgateway License configured: Community (LicensedModules: [base, auth], ExpiryDate: 2025-12-02)' observedGeneration: 7 reason: ValidLicense status: "True" type: Licensed ...

3. Detect invalid license states:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

   gwctl describe gateway -n example-ns mygateway Name: mygateway Namespace: example-ns ... Status: addresses: - type: IPAddress value: 100.126.1.2 conditions: - lastTransitionTime: "2025-11-26T15:40:15Z" message: Gateway is not licensed observedGeneration: 7 reason: Invalid status: "False" type: Accepted - lastTransitionTime: "2025-11-26T15:40:15Z" message: Gateway must be accepted first observedGeneration: 7 reason: Pending status: "False" type: Programmed - lastTransitionTime: "2025-11-26T15:40:15Z" message: No valid Airlock Microgateway license configured observedGeneration: 7 reason: InvalidLicense status: "False" type: Licensed ...

   The example above shows one invalid license state. Other invalid license states are also possible, for example InvalidConfigurationForLicense or ExpiredLicense. To determine the exact cause, check the values of the message and reason fields in the Licensed condition.

Verifying the GatewayParameters

Issues with GatewayParameters cause the Accepted and Programmed conditions to be False. This can happen in the following cases:

• The GatewayParameters resource cannot be resolved.
• The GatewayParameters contain unresolved references or other errors.

Use gwctl describe to inspect the reason of the Accepted condition for details:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

gwctl describe gateway -n example-ns mygateway
Name: mygateway
Namespace: example-ns
...
Status:
  addresses:
  - type: IPAddress
    value: 100.126.1.3
  conditions:
  - lastTransitionTime: "2025-11-27T07:34:45Z"
    message: |-
      Resolving GatewayParameters failed
      Missing referenced SessionHandling 'non-existent-resource'
    observedGeneration: 15
    reason: InvalidParameters
    status: "False"
    type: Accepted
...

Verifying all listeners

Listener specifications are part of the Gateway resource. gwctl describe lists all listeners with their names, their conditions, and the number of attached routes. Invalid listeners can cause requests to time out.

The gateway itself requires at least one valid listener. As long as at least one listener is valid, the gateway-level Accepted and Programmed conditions remain True. Therefore, a simple gwctl get is not sufficient to detect invalid listeners.

To verify that all listeners work as expected, perform the following steps:

1. Use gwctl describe gateway to list all listeners and their conditions.
2. Check each listener’s Accepted, Programmed, and ResolvedRefs conditions for status: “False”.
3. Use the reason and message fields to identify issues such as invalid secrets or unsupported protocols.
4. Ensure that all configured listeners are both accepted and programmed. Any listener not in this state indicates a configuration error and must be corrected.

At gateway level, the reason of the Accepted condition indicates that some listeners are invalid. Depending on the error, further details are available in the listener conditions.

The following example shows a gateway with three listeners, two of which are invalid (one with an invalid reference to a secret, one with an unsupported protocol). The message on the gateway’s Accepted condition indicates that only some, yet not all, listeners are valid:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

gwctl describe gateway -n example-ns mygateway
Name: mygateway
Namespace: example-ns
...
Status:
  addresses:
  - type: IPAddress
    value: 100.126.1.3
  conditions:
  - lastTransitionTime: "2025-11-27T08:09:20Z"
    message: Gateway has at least one valid listener
    observedGeneration: 21
    reason: ListenersNotValid
    status: "True"
    type: Accepted
...
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2025-11-27T08:08:44Z"
      message: Referenced certificate secret 'downstream-server-certificate-invalid'
        does not exist
      observedGeneration: 21
      reason: InvalidCertificateRef
      status: "False"
      type: ResolvedRefs
...
    - lastTransitionTime: "2025-11-27T08:08:44Z"
      message: Listener must have references resolved first
      observedGeneration: 21
      reason: Pending
      status: "False"
      type: Programmed
    name: https
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
  - attachedRoutes: 2
    conditions:
...
    - lastTransitionTime: "2025-11-27T08:08:44Z"
      message: 'Listener has an unsupported protocol, supported protocols are: [HTTP
        HTTPS microgateway.airlock.com/http3]'
      observedGeneration: 21
      reason: UnsupportedProtocol
      status: "False"
      type: Accepted
    - lastTransitionTime: "2025-11-27T08:08:44Z"
      message: Listener must be accepted first
      observedGeneration: 21
      reason: Pending
      status: "False"
      type: Programmed
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
...

• The Gateway is still considered accepted (type: Accepted with status: “True”) but reports ListenersNotValid.
• The individual listeners show the concrete problem causes — i.e., InvalidCertificateRef and UnsupportedProtocol; the corresponding message fields explain the root causes.
• The InvalidCertificateRef causes the affected listener’s ResolvedRefs condition to be False.
• The UnsupportedProtocol causes the affected listener’s Accepted condition to be False.
• For both invalid listeners, the Programmed condition is also False.

Checking for invalid routes

Microgateway exposes a dedicated AttachedRoutes condition on gateways to help detect issues on attached routes.

Use gwctl describe to inspect this condition and derive the next steps on route level.

Example with all routes valid:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

gwctl describe gateway -n example-ns mygateway
Name: mygateway
Namespace: example-ns
...
Status:
  addresses:
  - type: IPAddress
    value: 100.126.1.3
  conditions:
...
  - lastTransitionTime: "2025-11-27T07:44:19Z"
    message: All accepted and attached routes are valid
    observedGeneration: 23
    reason: AttachedRoutesValid
    status: "True"
    type: AttachedRoutes
...

Example with one or more invalid routes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

gwctl describe gateway -n example-ns mygateway
Name: mygateway
Namespace: example-ns
...
Status:
  addresses:
  - type: IPAddress
    value: 100.126.1.3
  conditions:
...
  - lastTransitionTime: "2025-11-28T09:59:23Z"
    message: |
      Gateway has the following invalid routes attached:
        (HTTPRoute) example-ns/backend
    observedGeneration: 23
    reason: InvalidRoutes
    status: "False"
    type: AttachedRoutes
...

Check HTTPRoute resource conditions

After the gateway has been thoroughly inspected, analyze the relevant HTTPRoutes. Faulty routes can lead to the following problems:

• A route that is not accepted and thus behaves as if it does not exist. Requests that should match this route return HTTP 404.
• A partially invalid route or an invalid backend. Requests matching the invalid part of the route return HTTP 500.

Getting a status overview

1. Use gwctl get for an overview of defined HTTPRoutes including the numbers of parent references and the attached policies as well as the statuses of the Accepted condition, and the ResolvedRefs condition:

   1 2 3 4 5

   gwctl get httproutes -n example-ns -o wide NAMESPACE NAME HOSTNAMES PARENT REFS ACCEPTED RESOLVED AGE POLICIES example-ns backend None 1 True True 30m 3 example-ns backend-tls None 2 True True 30m 3 example-ns example-rt None 1 True True 30m 1

2. Use gwctl describe to show the routes’ specifications, their status information and their attached policies:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

   gwctl describe httproutes -n example-ns backend Name: backend Namespace: example-ns ... Status: parents: - conditions: - lastTransitionTime: "2025-11-28T10:20:28Z" message: All attached policies are accepted observedGeneration: 1 reason: AttachedPoliciesAccepted status: "True" type: AttachedPolicies ... controllerName: microgateway.airlock.com/gatewayclass-controller parentRef: group: gateway.networking.k8s.io kind: Gateway name: my-gateway sectionName: http DirectlyAttachedPolicies: Type Name ---- ---- AccessControlPolicy.microgateway.airlock.com example-ns/ac-policy ContentSecurityPolicy.microgateway.airlock.com example-ns/cs-policy EnvoyExtensionPolicy.microgateway.airlock.com example-ns/ext-policy ...

   Notice

   • A route may be attached to more than one target.
   • Routes managed by the Microgateway operator have the attribute controllerName set to microgateway.airlock.com/gatewayclass-controller in the status overview.

Verifying route attachement

When checking routes for errors, perform the following steps:

1. Confirm that the number of parent references matches the expected value for each route.
2. Compare that with the number of HTTPRoutes reported by gwctl get gateway -o wide.
3. Look for unexpected discrepancies.

If a route contains an incorrect parent reference, the following issues will occur:

• The target gateway does not register it.
• The Microgateway operator does not reconcile it.
• Conditions and status information remain unchanged.
• observedGeneration values differ from the route’s generation.

gwctl describe reports such cases in the Analysis section:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

gwctl describe httproutes -n example-ns my-dangling-route                                                 
Name: my-dangling-route
Namespace: example-ns
...
Status:
...
Analysis:
- HTTPRoute(.gateway.networking.k8s.io) "example-ns/my-dangling-route" references
  a non-existent Gateway(.gateway.networking.k8s.io) "example-ns/some-non-existing-gw"
Events: <none>

Other inconsistencies between an HTTPRoute’s parent reference and the referenced gateway appear in the route’s Accepted condition. For example, a non-matching sectionName:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

gwctl describe httproutes -n example-ns backend
Name: backend
Namespace: example-ns
...
Status:
  parents:
  - conditions:
...
    - lastTransitionTime: "2025-11-28T13:19:37Z"
      message: ParentRef section name does not match any listener name on the Gateway
      observedGeneration: 2
      reason: NoMatchingParent
      status: "False"
      type: Accepted
...

Checking faulty references

References from an HTTPRoute to other objects (besides its parent) are another common error source. The ResolvedRefs condition indicates faulty references.

Use gwctl describe to inspect this condition:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

gwctl describe httproutes -n example-ns backend
Name: backend
Namespace: example-ns
...
Status:
  parents:
  - conditions:
...
    - lastTransitionTime: "2025-11-28T13:24:57Z"
      message: Service example-ns/non-existent-service not found.
      observedGeneration: 3
      reason: BackendNotFound
      status: "False"
      type: ResolvedRefs

Identifying invalid policies

Use gwctl describe to list the policies attached to an HTTPRoute:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

gwctl describe httproutes -n example-ns backend
Name: backend
Namespace: example-ns
...
Status:
  parents:
  - conditions:
...
    - lastTransitionTime: "2025-11-28T13:37:55Z"
      message: |
        Traffic may be blocked since HTTPRoute has the following invalid policies attached:
          (AccessControlPolicy) example-ns/ac-policy
      observedGeneration: 4
      reason: InvalidPolicies
      status: "False"
      type: AttachedPolicies
...

Microgateway exposes the AttachedPolicies condition:

• The message lists the affected policies.
• The status is True if all attached policies are valid.
• The status is False if one or more invalid policies are attached.

Check policy resource conditions

Policies are the last type of resource to analyze for errors or inconsistencies. Errors in policies cause affected requests to fail with HTTP 500.

Getting a status overview

1. Use gwctl get to list policies.

   In the following example output, targets are omitted for readability:

   1 2 3 4 5 6

   gwctl get policies -o wide -n example-ns NAMESPACE NAME KIND TARGET(S) POLICY TYPE ACCEPTED AGE example-ns ac-policy AccessControlPolicy.microgateway.airlock.com ... Direct True 3h36m example-ns backend-policy BackendTLSPolicy.gateway.networking.k8s.io ... Direct True 3h36m example-ns cs-policy ContentSecurityPolicy.microgateway.airlock.com ... Direct True 3h36m example-ns cr-policy CustomResponsePolicy.microgateway.airlock.com ... Direct True 3h36m

   The output shows the policies’ names, kinds, targets and types, as well as the statuses of their Accepted conditions.

   The policy kind defines the type of attachment (Direct or Inherited) and the target kinds a policy can attach to.

   Microgateway currently only supports policies of the Direct type.

2. To get more details on a policy, use gwctl describe with the correct kind of the policy to get as much detail as possible:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

   gwctl describe accesscontrolpolicy -n example-ns ac-policy apiVersion: microgateway.airlock.com/v1alpha1 kind: AccessControlPolicy metadata: ... name: ac-policy namespace: example-ns ... status: ancestors: - ancestorRef: group: gateway.networking.k8s.io kind: HTTPRoute name: backend conditions: - lastTransitionTime: "2025-11-28T14:28:47Z" message: AccessControlPolicy is accepted observedGeneration: 3 reason: Accepted status: "True" type: Accepted controllerName: microgateway.airlock.com/gatewayclass-controller ...

   Notice

   • A policy may be attached to more than one target.
   • Policies managed by the Microgateway operator have the attribute controllerName set to microgateway.airlock.com/gatewayclass-controller in the status overview.

Verifying policy attachement

1. Use gwctl describe to list the specified targets in the specification section and the resolved ancestors in the status section:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

   gwctl describe accesscontrolpolicy -n example-ns ac-policy apiVersion: microgateway.airlock.com/v1alpha1 kind: AccessControlPolicy metadata: ... generation: 2 ... name: ac-policy namespace: example-ns ... spec: ... targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: backend-one - group: gateway.networking.k8s.io kind: HTTPRoute name: backend-two status: ancestors: - ancestorRef: group: gateway.networking.k8s.io kind: HTTPRoute name: backend-one conditions: - lastTransitionTime: "2025-11-28T14:28:47Z" message: AccessControlPolicy is accepted observedGeneration: 2 ... - ancestorRef: group: gateway.networking.k8s.io kind: HTTPRoute name: backend-two conditions: ...

2. As policies may reference one or several targets, perform the following steps to verify that all specified targets are resolved:

   1. Inspect spec.targetRefs for the list of intended targets.
   2. Inspect status.ancestors for the resolved ancestors.
   3. Ensure that each targetRef has a corresponding ancestorRef.
   4. Compare the policy’s generation attribute with at least one condition’s observedGeneration attribute to ensure the Microgateway controller has updated the status.

   If more than one policy of the same kind reference the same target, the policies are marked as conflicting and only one of them has the status of its Attached condition set to True and is attached to the target. The other policies have the statuses of their Accepted conditions set to False and are not attached to the target.

3. Use gwctl get policies to list all policies.

   In the following example output, the policy backend-policy-2 is not accepted:

   1 2 3 4 5 6 7

   gwctl get policies -o wide -n example-ns NAMESPACE NAME KIND TARGET(S) POLICY TYPE ACCEPTED AGE example-ns ac-policy AccessControlPolicy.microgateway.airlock.com ... Direct True 3h36m example-ns backend-policy BackendTLSPolicy.gateway.networking.k8s.io ... Direct True 3h36m example-ns backend-policy-2 BackendTLSPolicy.gateway.networking.k8s.io ... Direct False 3h36m example-ns cs-policy ContentSecurityPolicy.microgateway.airlock.com ... Direct True 3h36m example-ns cr-policy CustomResponsePolicy.microgateway.airlock.com ... Direct True 3h36m

4. To get more details on a policy, use gwctl describe.

   The following example output reveals that the policy conflicts with another BackendTLSPolicy:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

   gwctl describe backendtlspolicies -n example-ns backend-policy-2 apiVersion: gateway.networking.k8s.io/v1 kind: BackendTLSPolicy metadata: ... name: backend-policy-2 namespace: example-ns ... status: ancestors: - ancestorRef: group: gateway.networking.k8s.io kind: Gateway name: airlock-microgateway conditions: ... - lastTransitionTime: "2025-11-28T15:03:10Z" message: 'BackendTLSPolicy is conflicting with other policies for this ancestor: [backend-policy]' observedGeneration: 1 reason: Conflicted status: "False" type: Accepted ...

Checking faulty references

Policies often contain many references and are therefore particularly susceptible to reference errors. Faulty references cause the status of the Accepted condition to be False, and the status of the ResolvedRefs condition (if present) to be False. The condition message contains details about the problem.

The following example shows an AccessControlPolicy with a faulty reference to an OIDCRelyingParty:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

gwctl describe accesscontrolpolicy -n example-ns ac-policy
apiVersion: microgateway.airlock.com/v1alpha1
kind: AccessControlPolicy
metadata:
...
  name: ac-policy
  namespace: example-ns
...
status:
  ancestors:
  - ancestorRef:
      group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: backend
    conditions:
    - lastTransitionTime: "2025-11-28T13:37:55Z"
      message: |-
        Resolving AccessControlPolicy failed, traffic to target will be blocked
        Missing referenced OIDCRelyingParty 'my-missing-oidc-rp'
      observedGeneration: 2
      reason: Invalid
      status: "False"
      type: Accepted
    controllerName: microgateway.airlock.com/gatewayclass-controller
...

Check events

Kubernetes events provide useful context for understanding issues.

Use kubectl with filters to restrict events to specific namespaces, resources, or types:

1
2
3
4
5

# Get all warnings across the cluster
kubectl events -A --types Warning

# Get only events for a gateway in a particular namespace
kubectl -n example-ns events --for gateway/<my-gateway>

Check logs

Logs from the Operator, the Engine or the Session Agent often reveal the root cause of unexpected behavior. Start with the Operator logs, then inspect the logs of the Gateway pods.

Operator logs

1

kubectl -n airlock-microgateway-system logs -l app.kubernetes.io/name=microgateway-operator -f

Gateway logs

The Gateway and GatewayParameters Custom Resource determine where (namespace) and how many pods are created. Each Gateway pod always contains an airlock-microgateway-engine container. Only when session handling is configured, the Gateway pod additionally contains an airlock-microgateway-session-agent container.

The commands below stream logs for all pods belonging to a Gateway (selected via label), choose the relevant container, and filter/format JSON log lines using jq.

Engine container: Access log

1
2
3
4

kubectl -n <airlock-gateway-namespace> logs \
  -l gateway.networking.k8s.io/gateway-name=<my-gateway> \
  -c airlock-microgateway-engine -f \
  | jq 'select(.log.logger=="access")'

Engine container: Application log (everything except access log)

1
2
3
4

kubectl -n <airlock-gateway-namespace> logs \
  -l gateway.networking.k8s.io/gateway-name=<my-gateway-name> \
  -c airlock-microgateway-engine -f \
  | jq 'select(.log.logger!="access")'

Session Agent container logs (only if session handling is enabled)

1
2
3
4

kubectl -n <airlock-gateway-namespace> logs \
  -l gateway.networking.k8s.io/gateway-name=<my-gateway-name> \
  -c airlock-microgateway-session-agent -f \
  | jq