Kubernetes api server dropped requests metric not visbile?

9/25/2018

I am trying to troubleshoot what should be the correct value for the --max-requests-inflight option to kube-apiserver. The reason I am doing this is that two of my four clusters are coming back with random 503's in response to calls to the REST api and I suspect this is the reason. The EC2 instances are the same in the other clusters and monitoring shows the instances are not overloaded.

However, to verify that this is the reason I need to see the dropped requests metric that was added to the api server, back in v1.8 if I am reading the history correctly.

I cannot find this anywhere in the metrics that are visible when using kubectl proxy and hitting the endpoint with a browser.

What am I doing wrong?

-- Juan Jimenez
api
kubernetes
metrics

1 Answer

9/26/2018

As @Rico mentioned, your link is broken. Going further to the --max-requests-inflight flag which is responsible for throttling API requests. This flag will allow limiting the number of API calls processed in parallel. There seems to be no recommendation in the official documentation, so I would try adjusting the value from 400 (default value specified in the documentation) and find a good balance. Note that too high number will result in killing the api-server because of Out of Memory, and too low value will result in API request-limit-exceed errors (there are examples of using this flag on around 2000 in a +700 pods cluster).

-- aurelius
Source: StackOverflow