The services in my kubernetes cluster are not reachable. When I checked the endpoints are not available for services. On checking further I found that the endpoint keep changing between some value and null. As suggested in other posts I checked the label selector. The pods are up. The containers are listening on the target port of the service. What else is going on here.
root@ak-host:/root# kubectl get endpoints
NAME ENDPOINTS AGE
Service1 2m
Service2 172.17.83.57:8006 2m
Service3 172.17.83.46:8082 2m
Service4 2m
Service5 2m
Service6 2m
Further update. The kube-controller-manager logs show the below error. E0810 20:02:21.887677 10451 nodecontroller.go:771] Error updating node : client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint.
Apparently the kube-apiserver in my machine had been using an argument '--etcd-servers=http://x.x.x.x:14001'. In my setup ha proxy would listen on 14001 and then forward it to 4001 where etcd would listen. Changing the init file for kube-apiserver to use port 4001 directly solved the issue in my case. Still not sure how haproxy was hindering the communication but now things work as expected.