I'm reading the Kubernetes docs on Reserve Compute Resources for System Daemons, and it says "Be extra careful while enforcing system-reserved reservation since it can lead to critical system services being CPU starved, OOM killed, or unable to fork on the node."
I've seen this warning in a few places, and I'm having a hard time understanding the practical implication.
Can someone give me a scenario in which enforcing system-reserved reservation would lead to system services being starved, etc, that would NOT happen if I did not enforce it?
You probably have at least a few things running on the host nodes outside of Kubernetes' view. Like systemd, some hardware stuffs, maybe sshd. Minimal OSes like CoreOS definitely have a lot less, but if you're running on a more stock OS image, you need to leave room for all the other gunk that comes with them. Without leaving RAM set aside, the Kubelet will happily use it all up and then when you go to try and SSH in to debug why your node has gotten really slow and unstable, you won't be able to.