We have a Kubernetes 1.7.8 clusters deployed with Kops 1.7 in HA with three masters. The cluster has 10 nodes and around 400 pods.
The cluster has heapster, prometheus, and ELK (collecting logs for some pods).
We are seeing a very high activity in the masters, over 90% of CPU used by the api-server.
Checking prometheus numbers we can see that near 5000 requests to the kube-apiserver are WATCH verbs, the rest are less than 50 request (GET, LIST, PATCH, PUT).
Almost all requests are reported with client "Go-Http-client/2.0" (the default User Agent for the Go HTTP library).
Is this a normal situation?
How can we debug which are the pods sending these requests? (How can we add the source IP to the kube-apiserver logs?)
[kube-apiserver.manifest][1]
Thanks, Charles
[1]: https://pastebin.com/nGxSXuZbRegarding the Kubernetes architecture this is a normal behavior because all kubernetes cluster components are calling the api-server to watch for changes.
That is why you have more than 5000 WATCH entries in your logs. Please take a look how the kubernetes cluster is managed by kube api server and how the master-node comunication is organized