Readiness probe failed: timeout: failed to connect service ":8080" within 1s

7/23/2020

I am trying to build and deploy microservices images to a single-node Kubernetes cluster running on my development machine using minikube. I am using the cloud-native microservices demo application Online Boutique by Google to understand the use of technologies like Kubernetes, Istio etc.

Link to github repo: microservices-demo

I have followed all the installation process to locally build and deploy the microservices, and am able to access the web frontend through my browser. However, when I click on any of the product images say, I see this error page.

HTTP Status: 500 Internal Server Error

On doing a check using kubectl get pods I realize that one of my pods( Recommendation service) has status CrashLoopBackOff. Running kubectl describe pods recommendationservice-55b4d6c477-kxv8r:

Namespace:    default
Priority:     0
Node:         minikube/192.168.99.116
Start Time:   Thu, 23 Jul 2020 19:58:38 +0530
Labels:       app=recommendationservice
              app.kubernetes.io/managed-by=skaffold-v1.11.0
              pod-template-hash=55b4d6c477
              skaffold.dev/builder=local
              skaffold.dev/cleanup=true
              skaffold.dev/deployer=kubectl
              skaffold.dev/docker-api-version=1.40
              skaffold.dev/run-id=49913ced-e8df-40a7-9336-a227b56bcb5f
              skaffold.dev/tag-policy=git-commit
Annotations:  <none>
Status:       Running
IP:           172.17.0.14
IPs:
  IP:           172.17.0.14
Controlled By:  ReplicaSet/recommendationservice-55b4d6c477
Containers:
  server:
    Container ID:   docker://2d92aa966a82fbe58c8f40f6ecf9d6d55c29f8081cb40e0423a2397e1419350f
    Image:          recommendationservice:2216d526d249cc8363129aed9a09d752f9ad8f458e61e50a2a99c59d000606cb
    Image ID:       docker://sha256:2216d526d249cc8363129aed9a09d752f9ad8f458e61e50a2a99c59d000606cb
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Thu, 23 Jul 2020 21:09:33 +0530
      Finished:     Thu, 23 Jul 2020 21:09:53 +0530
    Ready:          False
    Restart Count:  29
    Limits:
      cpu:     200m
      memory:  450Mi
    Requests:
      cpu:      100m
      memory:   220Mi
    Liveness:   exec [/bin/grpc_health_probe -addr=:8080] delay=0s timeout=1s period=5s #success=1 #failure=3
    Readiness:  exec [/bin/grpc_health_probe -addr=:8080] delay=0s timeout=1s period=5s #success=1 #failure=3
    Environment:
      PORT:                          8080
      PRODUCT_CATALOG_SERVICE_ADDR:  productcatalogservice:3550
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-sbpcx (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-sbpcx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-sbpcx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Pulled     44m (x15 over 74m)     kubelet, minikube  Container image "recommendationservice:2216d526d249cc8363129aed9a09d752f9ad8f458e61e50a2a99c59d000606cb" already present on machine
  Warning  Unhealthy  9m33s (x99 over 74m)   kubelet, minikube  Readiness probe failed: timeout: failed to connect service ":8080" within 1s
  Warning  BackOff    4m25s (x294 over 72m)  kubelet, minikube  Back-off restarting failed container

In Events, I see Readiness probe failed: timeout: failed to connect service ":8080" within 1s. What is the reason and how can I resolve this? Thanks for the help!

-- Saranya Gupta
docker-container
kubernetes
microservices

1 Answer

7/25/2020

Answer

The timeout of the Readiness Probe (1 second) was too short.

More Info

The relevant Readiness Probe is defined such that /bin/grpc_health_probe -addr=:8080 is run inside the server container.

You would expect a 1 second timeout to be sufficient for such a probe but this is running on Minikube so that could be impacting the timeout of the probe.

-- Serge
Source: StackOverflow