Is there a way to change the logging level on a running cluster, specifically for a single system component such as kube-proxy?
I see some discussion related to this:
https://github.com/kubernetes/test-infra/pull/4311 and it seems like some sort of a mechanism was put in place, but it is not clear to me how to use this mechanism.
This question came about because we are troubleshooting connections to a NodePort service (which should go through kube-proxy), and at the default level of --v=2 kube-proxy doesn't seem to log any of the connections that it proxies, so seeking to increase it.
it seems like some sort of a mechanism was put in place, but it is not clear to me how to use this mechanism
I didn't see anything in that PR that would lead me to believe there is a dynamic (that is: without terminating kube-proxy) mechanism for altering log levels. There is no dynamic logging adjustment mechanism that I'm aware of in any of the kubernetes components.
However, kube-proxy (traditionally) runs in a docker container just like any other Pod, and thus is subject to being restarted on termination. So just update its --v in the manifest, kill the container (the one without Pod in its name), and kubelet will start kube-proxy back up, now with the new --v level.
(which should go through kube-proxy)
Just for clarity, kube-proxy only manages the iptables rules in its default configuration, and so no traffic flows through it that I'm aware of. That's actually why it's safe to just restart kube-proxy at will.
You can examine the rules it puts into place with the regular iptables -t nat -L command, and kube-proxy is even helpful enough to add comments to the rules, showing which kubernetes service they represent.