Pod is not ready

It might be possible that the following problem could be observed after labeling a working Pod with sidecar.microgateway.airlock.com/inject: true.

  • The Pod is in running state but not ready.
  • The old Pod is still handling the requests and the rollover is not completed.

The examples depend on the fact that the web application Pod was running perfectly before labeling it.

Verify the Pod's state

Follow the instructions below to verify the Pod's state.

  1. Use kubectl describe pod to retrieve the status.
  2. kubectl describe pod webapp-84b785f954-r49wt
    Readiness Gates:
      Type                                           Status
      microgateway.airlock.com/engine-config-ready   False
      Type                                           Status
      Initialized                                    True
      Ready                                          False
      ContainersReady                                True
      PodScheduled                                   True 
      microgateway.airlock.com/engine-config-ready   False

    The Readiness Gates microgateway.airlock.com/engine-config-ready indicates whether the Microgateway Engine container is ready or not.

  3. Log messages containing initial fetch timed are logged by the Microgateway Engine container when it is not ready.
  4. kubectl logs webapp-84b785f954-r49wt -c airlock-microgateway-engine
    [2022-09-13 11:05:12.130][1][warning][config] [external/envoy/source/common/config/grpc_subscription_impl.cc:118] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.listener.v3.Listener

Other observed behaviors

Depending on where you look first, you might face other issues. The examples below illustrate how the same issue could look when using other commands.

  • The old and new Deployment are in Running state.
  • kubectl get deployment 
    NAME             READY   UP-TO-DATE   AVAILABLE   AGE 
    webapp           1/1     1            1           29m
    kubectl get pods  
    NAME                             READY   STATUS      RESTARTS   AGE 
    webapp-849f69665c-hw4t4          1/1     Running     0          29m 
    webapp-84b785f954-r49wt          2/2     Running     0          29m
  • The command kubectl get pods -o wide shows that the Pod is not yet ready.
  • kubectl get pods -o wide 
    NAME                             READY   STATUS      RESTARTS   AGE     IP             NODE           NOMINATED NODE   READINESS GATES 
    webapp-849f69665c-hw4t4          1/1     Running     0          32m        worker-0   <none>           <none> 
    webapp-84b785f954-r49wt          2/2     Running     0          32m        worker-1   <none>           0/1
  • The Readiness and Liveness probes failed.
  • kubectl describe pod webapp-84b785f954-r49wt
      Type     Reason     Age                From               Message 
      ----     ------     ----               ----               -------
    Warning  Unhealthy  50m (x2 over 50m)  kubelet            Liveness probe failed: Get "": dial tcp connect: connection refused 
    Warning  Unhealthy  50m (x2 over 50m)  kubelet            Readiness probe failed: Get "": dial tcp connect: connection refused

Possible reasons

The following reasons could cause the observed behavior.

  1. No Custom Resource SidecarGateway is associated with the Pod. Check the following:
  2. If the Readiness and Liveness probes for the Microgateway Engine have failed.
  3. Ensure that a CR SidecarGateway exists and matches the Web application Pod labels.
  4. Most likely, no CR SidecarGateway matching the Pod. Follow the instructions in CR SidecarGateway to correct this.
  1. The referenced Custom Resource does not exist. Check the following:
  2. Ensure that the referenced Custom Resource exists.
  3. Check the status of the Custom Resource SidecarGateway:
  4. kubectl get sidecargateways.microgateway.airlock.com  
    NAME    STATUS    AGE 
    webapp  Error   3m25s
    kubectl describe sidecargateways.microgateway.airlock.com webapp 
        Last Transition Time:  2022-09-13T12:48:24Z 
        Message:               Missing referenced ContentSecurity 'webapp' 
        Reason:                ConfigResourcesMissing 
        Status:                False 
        Type:                  ConfigReferencesLoaded 
        Name:  webapp-849f69665c-hw4t4 
        Name:  webapp-84b785f954-r49wt 
      Status:  Error 
    Events:    <none>
  5. The status shows if the SidecarGateway is ready or not. In this example, the referenced ContentSecurity is missing. Follow the instructions in CR SidecarGatewayto correct this.