I am new to K8s. I am researching benefits of using k8s on GKE/AKS over Azure VM Scalesets used in our system.
For k8s, let's say we deploy worker application to 5 nodes cluster, each node can runs 10 pods. Each node is 16 core, 64G memory VM.
My question is, since we are paying cloud service provider by node, not by pod, why we want to scale down pod on the node at all?
Does it make more sense to horizontally scale up and down nodes only, while having maximum pods on each node?
Does 10 pods, 20 pods, 30 pods, 40 pods, 50 pods makes more sense, meanwhile 11 pods, 21 pods, 31 pods, 41 pods etc sounds wasting resources we paid?
I must missed key points of doing pods scaling. Please point them out. Thanks.
If all your pods are all identical, then yes, it likely makes sense to scale them only in node-sized equivalents (but maybe in that case you want to actually make sure the nodes can scale in increments that make sense to your workload size -- e.g. do you really need another 32 or 64 or 96 cores worth of your application every time? This is one of the things that the new Google batch for Kubernetes product tries to address -- including rightsizing of machines).
But think if you have a heterogeneous set of workloads (far more likely with k8s!) -- then one of the advantages of k8s is that you can binpack different workloads onto the same node.
Imagine if one workload needs a lot of RAM, but not much CPU, and another workload needs a lot of CPU and not a lot of RAM -- you wouldn't want them to both scale up in one-node increments, you'd want the pods to scale with the demands on each application, and the nodes to scale when you can no longer binpack onto the existing machines.