I'm using Cluster-autoscaler (v1.18.3) on Amazon EKS (Cluster version 1.18). Every time we've an increase in load the cluster-autoscaler adds new nodes to deal with the extra load. But when the load decreases cluster-autoscaler need to reduce the number of nodes back to minimum. Since I've set on all the Statefulsets pod template the annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" I do not expect the cluster-autoscaler to remove any node that contains any pods having this annotation. I can also see cluster-autoscaler logs showing 1 cluster.go:168] Fast evaluation: node ip-172-30-59-87.eu-west-2.compute.internal cannot be removed: pod annotated as not safe to evict present: jenkins-0 but still cluster-autoscaler removes this exact same nodes (that contain the statefulset pods).

I've noticed cluster-autoscaler is always removing oldest nodes from the cluster when scaling in.

I also tried setting node annotations of "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true" on the nodes which contain those important Jenkins statefulset pods. But still cluster-autoscaler will always remove the oldest nodes in the cluster. No matter what node / pod annotations I set it will always remove the oldest nodes.

The issue

I don't want nodes that contain Statefulset pods which I've annotated as "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" to be removed. The new nodes that were launched to deal with the load should be the ones that should be removed.

I also do not expect cluster-autoscaler to remove a node that has annotation "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true" to be removed.

The oldest nodes are the ones that are always removed. Why is this happening? I've opened an issue/bug 3 weeks before opening this post https://github.com/kubernetes/autoscaler/issues/3978 but this hasn't been commented or picked up by anyone.

How I've setup cluster-autoscaler?

Deployed cluster autoscaler using AWS Documentation https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html or https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md.

How to produce the issue?

Ensure you've setup an EKS cluster with 2 node groups (One per Az) with 1 node min in each node group. 1. Install Jenkins statefulset using helm chart https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins. 2. Annotate pods in values file

...
controller:
  podAnnotations:
    "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
...

Note: I've just mentioned the podAnnotations here as the values file is more than 200 lines of code setting different things like pvc, config as code, plugins etc.

3. Deploy cluster-autoscaler from AWS documentation.
4. Now to produce some load on the cluster create a simple Deployment of Nginx (3GBRAM, 2CPU) & Scale out to 50 replicas. Cluster Autoscaler will add new EC2 nodes.
5. Now scale down replicas to 0 and wait for 10 mins for Cluster Autoscaler to scale down. Cluster Autoscaler will log output like 1 cluster.go:168] Fast evaluation: node ip-172-30-59-87.eu-west-2.compute.internal cannot be removed: pod annotated as not safe to evict present: jenkins-0 but still remove this exact node.

Tried Configurations

Cluster-autoscaler images tried: 1.18.3, 1.18.2 on Kubernetes 1.18 cluster. Also tried images 1.19.2 on Kubernetes 1.19 cluster with exact same results.

Ran cluster-autoscaler with configuration:

spec: 
  containers: 
  - command: 
    - ./cluster-autoscaler 
    - --v=4 
    - --stderrthreshold=info 
    - --cloud-provider=aws 
    - --skip-nodes-with-local-storage=false 
    - --expander=least-waste 
    - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/xxxmyclusternamexxx 
    - --balance-similar-node-groups 
    - --skip-nodes-with-system-pods=false

also tried:

spec: 
  containers: 
   - command: 
     - ./cluster-autoscaler 
     - --v=4 
     - --stderrthreshold=info 
     - --cloud-provider=aws 
     - --expander=least-waste 
     - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/xxxxmyclusternamexxxxx 
     - --balance-similar-node-groups