I have a Kubernetes cluster running various apps with different machine types (ie. cpu-heavy, gpu, ram-heavy) and installed cluster-autoscaler (CA) to manage the Auto Scaling Groups (ASG) using auto-discovery.
Kubernetes version: EKS 1.11
Cluster Autoscaler: v1.13.2
I have configured my ASGs such that they contain the appropriate CA tags. These ASGs are usually scaled down to 0 nodes, and will be scaled up according to workload. Going into the CA logs, I have also verified that the CA acknowledges the existence of the ASGs. However, whenever I try to create pods with nodeSelectors, the CAs doesn't scale up the corresponding ASG and responds with "predicate failed: nodeSelector(s) did not match". I have added the appropriate node labels to the ASG's tags as well.
Could it be a limitation due to scaling up from 0 nodes?
Can't seem to find similar problems online. Any feedback would help! Thanks!
I have a much simpler setup here. I have just one group of ASG. The minimum amount is 1 and 10 maximum.
Maybe auto discovery is not working the way it should. It is the best approach, but it is not working. I would give a try on the example below to manage multiple ASG.
I have the same doubt as you, I am not sure if autoscaler works properly ranging from 0 nodes.
Update: CA Documentation update
I had the incorrect ASG tag. Ensure that you have k8s.io/cluster-autoscaler/node-template/label/some-label to ensure that CA can see your node labels when the ASG is scaled down to 0. Currently, CA doesn't create node labels based on ASG tags, but this feature is in the pipeline.