Pod is not ready

It might be possible that the following problem could be observed after annotating a working Pod with sidecar.microgateway.airlock.com/inject: true.

  • The Pod is in running state but not ready.
  • The old Pod is still handling the requests and the rollover is not completed.

The examples depend on the fact that the web application Pod was running perfectly before annotating it.

Verify the Pod's state

Follow the instructions below to verify the Pod's state.

  1. Use kubectl describe pod to retrieve the status.
    2. copy
    kubectl describe pod webapp-84b785f954-r49wt
 ...
Readiness Gates:
  Type                                           Status
  microgateway.airlock.com/engine-config-ready   False
Conditions:
  Type                                           Status
  Initialized                                    True
  Ready                                          False
  ContainersReady                                True
  PodScheduled                                   True 
  microgateway.airlock.com/engine-config-ready   False

    The Readiness Gates microgateway.airlock.com/engine-config-ready indicates whether the Microgateway Engine container is ready or not.

  2. Log messages containing initial fetch timed are logged by the Microgateway Engine container when it is not ready.
    3. copy
    kubectl logs webapp-84b785f954-r49wt -c airlock-microgateway-engine
...
[2022-09-13 11:05:12.130][1][warning][config] [external/envoy/source/common/config/grpc_subscription_impl.cc:118] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.listener.v3.Listener
...

Other observed behaviors

Depending on where you look first, you might face other issues. The examples below illustrate how the same issue could look when using other commands.

  • The old and new Deployment are in Running state.
    • copy
    kubectl get deployment 
NAME             READY   UP-TO-DATE   AVAILABLE   AGE 
webapp           1/1     1            1           29m
    copy
    kubectl get pods  
NAME                             READY   STATUS      RESTARTS   AGE 
webapp-849f69665c-hw4t4          1/1     Running     0          29m 
webapp-84b785f954-r49wt          2/2     Running     0          29m
  • The command kubectl get pods -o wide shows that the Pod is not yet ready.
    • copy
    kubectl get pods -o wide 
NAME                             READY   STATUS      RESTARTS   AGE     IP             NODE           NOMINATED NODE   READINESS GATES 
webapp-849f69665c-hw4t4          1/1     Running     0          32m     10.43.1.173        worker-0   <none>           <none> 
webapp-84b785f954-r49wt          2/2     Running     0          32m     10.43.2.173        worker-1   <none>           0/1
  • The Readiness and Liveness probes failed.
    • copy
    kubectl describe pod webapp-84b785f954-r49wt
...
Events: 
  Type     Reason     Age                From               Message 
  ----     ------     ----               ----               -------
...
Warning  Unhealthy  50m (x2 over 50m)  kubelet            Liveness probe failed: Get "http://10.43.2.173:19001/ready": dial tcp 10.43.2.173:19001: connect: connection refused 
Warning  Unhealthy  50m (x2 over 50m)  kubelet            Readiness probe failed: Get "http://10.43.2.173:19001/ready": dial tcp 10.43.2.173:19001: connect: connection refused

Possible reasons

The following reasons could cause the observed behavior.

  1. No Custom Resource SidecarGateway is associated with the Pod. Check the following:
  2. If the Readiness and Liveness probes for the Microgateway Engine have failed.
  3. Ensure that a CR SidecarGateway exists and matches the Web application Pod labels.
  4. Most likely, no CR SidecarGateway matching the Pod. Follow the instructions in CR SidecarGateway to correct this.
  1. The referenced Custom Resource does not exist. Check the following:
  2. Ensure that the referenced Custom Resource exists.
  3. Check the status of the Custom Resource SidecarGateway:
    4. copy
    kubectl get sidecargateways.microgateway.airlock.com  
NAME    STATUS    AGE 
webapp  Error   3m25s
    copy
    kubectl describe sidecargateways.microgateway.airlock.com webapp 
...
Status: 
  Conditions: 
    Last Transition Time:  2022-09-13T12:48:24Z 
    Message:               Missing referenced ContentSecurity 'webapp' 
    Reason:                ConfigResourcesMissing 
    Status:                False 
    Type:                  ConfigReferencesLoaded 
  Pods: 
    Name:  webapp-849f69665c-hw4t4 
    Name:  webapp-84b785f954-r49wt 
  Status:  Error 
Events:    <none>
  4. The status shows if the SidecarGateway is ready or not. In this example, the referenced ContentSecurity is missing. Follow the instructions in CR SidecarGatewayto correct this.