Kubernetes Control Plane communication
I am researching the K8s architecture, focusing on the flow of pod spinning in the system.
I was wondering how (that is, who is initiating the communication) and when the different components of the control plane communicate with each other.
I have followed the excellent talk of Jérôme Petazzoni at LISA 2019 (here) to understand the architecture of the control plane, and read the concepts on kubernetes.io.
However, I still haven't found the answers to the following questions:
Who initiates the resource check of each node, in the documentation it is written:
Node objects track information about the Node's resource capacity: for example, the amount of memory available and the number of CPUs. Nodes that self-register report their capacity during registration. If you manually add a Node, then you need to set the node's capacity information when you add it.
However, there is no specification on when does it update at etcd, and who initiates the regular update (other than the heartbeat that updates the status of the node).
Also, when does the cache of the scheduler update?
- Who informs the different components about new pending requests? That is, how is the
controller-manager/scheduler"knows" when it suppose to do its job? Each request is written as a manifest inetcdby thekube-api-server, but these components aren't connected toetcddirectly.
Does that mean the API-Server needs to inform each component about each new request?
I have many possible answers, but not a concrete confirmation of the real process in current K8s architecture.
Similar Questions
1 Answer
Answering your questions:
Who initiates the resource check of each node?
The component responsible for that is "Node Status Manager" which is a sub-control loop of the "SyncLoop" which is a kubelet agent component.
The more detailed answer is in this article: Kubernetes Deep Dive: Kubelet:
As you can see, the core of
kubelet’s work is a control loop, namely: SyncLoop.For example, the Node Status Manager is responsible for responding to changes in the status of the
Node, and then collecting the status of theNodeand reporting it to theAPIServerthrough Heartbeat.
There is also a good diagram:
Answering second part:
Who informs the different components about new pending requests? That is, how is the controller-manager/scheduler "knows" when it suppose to do its job?
The components responsible for that are Kubernetes' controllers and Scheduler itself. Good examples and explanations are in this article: What happens when ... Kubernetes edition!,
Basically after Kubernetes verified the request (authentication, authorization, admission control stuff), it is saved to datastore (etcd), and then it's taken by initializers which may perform some additional logic on the resource (not always), after that it's visible via kube-server. Main part that may interest you is Control loops. They are constantly checking if a new request exists in a datastore, and if yes they are proceeding. Example - when you are deploying a new deployment:
- Deployments controller is taking a request - it will realise that there is no ReplicaSet record associated, it will roll-out new one
- ReplicaSets controller, like deployments controller, it will take a request and roll-out new pods
- Pods are ready but they are in pending state - now Scheduler (which is like previous controllers, listening constantly for new requests from the data store - it's de facto answer for your question) will find a suitable node and schedule a pod to a node. Now, kubelet agent on the node will create a new pod.
For more details I'd strongly suggest reading the earlier mentioned article - What happens when ... Kubernetes edition!.
Does that mean the API-Server needs to inform each component about each new request?
It works in a different way - the kube-apiserver is making requests to be visible, and controllers, which are loops, are detecting new requests and starting to proceed with them.