endpoints flapping in kubernetes

8/10/2016

The services in my kubernetes cluster are not reachable. When I checked the endpoints are not available for services. On checking further I found that the endpoint keep changing between some value and null. As suggested in other posts I checked the label selector. The pods are up. The containers are listening on the target port of the service. What else is going on here.

root@ak-host:/root# kubectl get endpoints
NAME     ENDPOINTS           AGE
Service1                     2m
Service2 172.17.83.57:8006   2m
Service3 172.17.83.46:8082   2m
Service4                     2m
Service5                     2m
Service6                     2m

Further update. The kube-controller-manager logs show the below error. E0810 20:02:21.887677 10451 nodecontroller.go:771] Error updating node : client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint.

-- Akilesh
containers
flanneld
kube-dns
kubernetes

1 Answer

8/30/2016

Apparently the kube-apiserver in my machine had been using an argument '--etcd-servers=http://x.x.x.x:14001'. In my setup ha proxy would listen on 14001 and then forward it to 4001 where etcd would listen. Changing the init file for kube-apiserver to use port 4001 directly solved the issue in my case. Still not sure how haproxy was hindering the communication but now things work as expected.

-- Akilesh
Source: StackOverflow