0 response code on prometheus reporting for WATCH operations against kubernetes apiserver

6/14/2020

We recently upgraded our kubernetes control plane from version 1.13 to 1.16

After doing that, we started seeing 0 (zero) response codes on the prometheus reporting dashboard for WATCH operations against the apiserver.

I tried running a kubectl command to watch a subset of pods with a specific label and received 200OK so I couldn't reproduce it from my end.

The metric we're using is

sum(rate(apiserver_request_count{code!~"^2.*
quot;
}[1m])) by (code, resource, subresource, verb)

I tried searching for any clue of what might be the cause and the only one I found is https://github.com/kubernetes/kubernetes/issues/16686

Is it something we should be worried about or is it just a nuance of the new apiserver?

Thanks for any help!

-- Erez Rabih
kubernetes
kubernetes-apiserver
prometheus

0 Answers