I'm pretty new to k8s. How does k8s deal with node resource changes? For example, if some of the nodes are VMs and their resource assignment (cpu & memory) changes over time, does k8s control plane constantly monitor the computation resources available to each node and update its records?
To be precise, what do you mean to 'changes at run time'? Did you mean that VM(node) is in cluster and you will turn off this VM, add new resources and turn on again?
If yes, please find some information here and some examples below:
Minikube
You are not able to change it. During creation of Minikube cluster, if you did not specify resources, it will automatically create cluster with Minikube name, 2 CPUs and 2048 RAM.
If you will specify Minikube resources for example:
$ sudo minikube start --cpus=4 --memory=8192 -p minikube-name
Kubeadm
Basically if mentioned VM is not Master, you just to have to turn off the VM, add resources and run it again. Kubeadm automatically will recognize that Node resources has been changed.
$ kubectl describe node <node-name>
$ kubectl describe node ubuntu18-slave
...
Addresses:
InternalIP: 10.156.0.16
Hostname: ubuntu18-slave
Capacity:
cpu: 2
ephemeral-storage: 9983232Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7652372Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 9200546596
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7549972Ki
pods: 110
...Added resources. Just turn off and turn on VM, without kubelet restart.
$ kubectl describe node ubuntu18-slave
...
Addresses:
InternalIP: 10.156.0.16
Hostname: ubuntu18-slave
Capacity:
cpu: 8
ephemeral-storage: 9983232Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 30875684Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 9200546596
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 30773284Ki
pods: 110
...Cloud
It depends from cloud providedr. I've tried to achieve this with GKE. I found only one way to do it. You just have to: * Create a new node_pool with specific resources * Add them to cluster * Drain old node * Delete old node
NOTE:
Please remember that before turn off VM (Node) you should drain node you want to turn off.
Monitoring
To achieve resources usage in current time, create history of usage you can use some OpenSource or Paid software for it. All depends on your needs
However I would recommended you to use Prometheus as it is one of the most popular and you can find many tutorials online.
Here you can find many useful information about metrics and prometheus.
Yes, you can add grafana to see these variations in more descriptive way or kubernetes dashboard can also be helpful.
You can also calculate the resource limits assigned to the pods of your applications, the nodes where they have been scheduled, thereby getting almost an approximate idea of the memory and cpu occupancy against the worker nodes.
I found this link very useful. To answer your question if K8s monitor resources, yes it does, as it has to make sure the resources required on a node are within the limits allowed for a resource, for e.g, CPU.