GCE: load balancer health check fails when routing traffic to pods

11/11/2016

I am following the steps at https://codelabs.developers.google.com/codelabs/cloud-hello-kubernetes and can successfully expose my pod to the outside world with a command like:

kubectl expose deployment hello --type="LoadBalancer"

I have set up a static IP and when I run

$ kubectl get services
NAME         CLUSTER-IP      EXTERNAL-IP       PORT(S)   AGE
kubernetes   10.111.xxx.x    <none>            443/TCP   13d
hello        10.111.xxx.xx   104.155.xxx.xxx   80/TCP    12d

Everything looks OK and works for a couple of days, but after a while the traffic from 104.155.xxx.xxx stops getting routed to my pod and I start getting errors like this when I check the load balancer:

Instance gke-k8-default-pool-xxxx is unhealthy for 104.155.xxx.xxx

This always happens after a few days. I have no clue what I am doing wrong.

-- emh
gcp
google-compute-engine
kubernetes
load-balancing

1 Answer

4/13/2017

The load-balancer functionality is provided by the underlying infrastructure(in your case is GCE), so this has barely no relation with kubernetes it self.

Instance gke-k8-default-pool-xxxx is unhealthy for 104.155.xxx.xxx

From the log you provide, I can only tell the instance(vm) in your GCE can't pass the health check you defined for ip 104.155.xxx.xxx. So there is several things you need to check:

  1. Did anything special happened for instance gke-k8-default-pool-xxxx?
  2. What health check you defined for 104.155.xxx.xxx (why would it down for this instance) ?

So you can choose to find out what the true reason above or just restart the instance gke-k8-default-pool-xxxx to check if it succeed again.

-- Crazykev
Source: StackOverflow