K
Q

Horizontal Pod Autoscaler replicas based on the amount of the nodes in the cluster

1/21/2020

Im looking for the solution that will scale out pods automatically when the nodes join the cluster and scale in back when the nodes are deleted. We are running WebApp on the nodes and this require graceful pod eviction/termination when the node is scheduled to be disconnected. I was checking the option of using the DaemonSet but since we are using Kops for the cluster rolling update it ignores DaemonSets evictions (flag "--ignore-daemionset" is not supported). As a result the WebApp "dies" with the node which is not acceptable for our application. The ability of HorizontalPodAutoscaler to overwrite the amount of replicas which are set in the deployment yaml could solve the problem. I want to find the way to change the min/maxReplicas in HorizontalPodAutoscaler yaml dynamically based on the amount of nodes in the cluster.

spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: MyWebApp
  minReplicas: "Num of nodes in the cluster"
  maxReplicas: "Num of nodes in the cluster"

Any ideas how to get the number of nodes and update HorizontalPodAutoscaler yaml in the cluster accordingly? Or any other solutions for the problem?

-- DenisTs
autoscaling
horizontal-pod-autoscaling
kops
kubernetes
kubernetes-pod

Similar Questions

Pull image from private docker registry in AWS EKS Autoscaler worker nodes
How to achieve zero cost and secure deployment option using docker?
use of kubectl log in readiness probe
cert-manager-webhook: FailedDiscoveryCheck, namespace hangs in termination
TCP exposed service in an Ingress Nginx works with ssl?
Kubernetes - calico IP_AUTODETECTION_METHOD impossible to set
Can't access kubernetes service of type LoadBalancer on AKS

1 Answer

1/22/2020

Have you tried usage of nodeSelector spec in daemonset yaml. So if you have nodeselector set in yaml and just before drain if you remove the nodeselector label value from the node the daemonset should scale down gracefully also same when you add new node to cluster label the node with custom value and deamonset will scale up.

This works for me so you can try this and confirm with Kops

First : Label all you nodes with a custom label you will always have on your cluster

Example:

kubectl label nodes k8s-master-1 mylabel=allow_demon_set  
kubectl label nodes k8s-node-1 mylabel=allow_demon_set
kubectl label nodes k8s-node-2 mylabel=allow_demon_set
kubectl label nodes k8s-node-3 mylabel=allow_demon_set

Then to your daemon set yaml add node selector.

Example.yaml used as below : Note added nodeselctor field

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      nodeSelector:
        mylabel: allow_demon_set
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

So nodes are labeled as below

$ kubectl get nodes --show-labels
NAME           STATUS   ROLES    AGE   VERSION   LABELS
k8s-master-1   Ready    master   9d    v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master-1,kubernetes.io/os=linux,mylable=allow_demon_set,node-role.kubernetes.io/master=
k8s-node-1     Ready    <none>   9d    v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-1,kubernetes.io/os=linux,mylable=allow_demon_set
k8s-node-2     Ready    <none>   9d    v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-2,kubernetes.io/os=linux,mylable=allow_demon_set
k8s-node-3     Ready    <none>   9d    v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-3,kubernetes.io/os=linux,mylable=allow_demon_set

Once you have correct yaml start the daemon set using it

$ kubectl create -f Example.yaml

$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
pod/fluentd-elasticsearch-jrgl6   1/1     Running   0          20s   10.244.3.19   k8s-node-3     <none>           <none>
pod/fluentd-elasticsearch-rgcm2   1/1     Running   0          20s   10.244.0.6    k8s-master-1   <none>           <none>
pod/fluentd-elasticsearch-wccr9   1/1     Running   0          20s   10.244.1.14   k8s-node-1     <none>           <none>
pod/fluentd-elasticsearch-wxq5v   1/1     Running   0          20s   10.244.2.33   k8s-node-2     <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   9d    <none>

NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR             AGE   CONTAINERS              IMAGES                                         SELECTOR
daemonset.apps/fluentd-elasticsearch   4         4         4       4            4           mylable=allow_demon_set   20s   fluentd-elasticsearch   quay.io/fluentd_elasticsearch/fluentd:v2.5.2   name=fluentd-elasticsearch

Then before draining a node we can just remove the custom label from node and the pod-should scale down gracefully and then drain the node.

$ kubectl label nodes k8s-node-3 mylabel-

Check the daemonset and it should scale down

ubuntu@k8s-kube-client:~$ kubectl get all -o wide
NAME                              READY   STATUS        RESTARTS   AGE     IP            NODE           NOMINATED NODE   READINESS GATES
pod/fluentd-elasticsearch-jrgl6   0/1     Terminating   0          2m36s   10.244.3.19   k8s-node-3     <none>           <none>
pod/fluentd-elasticsearch-rgcm2   1/1     Running       0          2m36s   10.244.0.6    k8s-master-1   <none>           <none>
pod/fluentd-elasticsearch-wccr9   1/1     Running       0          2m36s   10.244.1.14   k8s-node-1     <none>           <none>
pod/fluentd-elasticsearch-wxq5v   1/1     Running       0          2m36s   10.244.2.33   k8s-node-2     <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   9d    <none>

NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR             AGE     CONTAINERS              IMAGES                                         SELECTOR
daemonset.apps/fluentd-elasticsearch   3         3         3       3            3           mylable=allow_demon_set   2m36s   fluentd-elasticsearch   quay.io/fluentd_elasticsearch/fluentd:v2.5.2   name=fluentd-elasticsearch

Now again add the label to new node with same custom label when it is added to cluster and the deamonset will scale up

$ kubectl label nodes k8s-node-3 mylable=allow_demon_set

ubuntu@k8s-kube-client:~$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP            NODE           NOMINATED NODE   READINESS GATES
pod/fluentd-elasticsearch-22rsj   1/1     Running   0          2s      10.244.3.20   k8s-node-3     <none>           <none>
pod/fluentd-elasticsearch-rgcm2   1/1     Running   0          5m28s   10.244.0.6    k8s-master-1   <none>           <none>
pod/fluentd-elasticsearch-wccr9   1/1     Running   0          5m28s   10.244.1.14   k8s-node-1     <none>           <none>
pod/fluentd-elasticsearch-wxq5v   1/1     Running   0          5m28s   10.244.2.33   k8s-node-2     <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   9d    <none>

NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR             AGE     CONTAINERS              IMAGES                                         SELECTOR
daemonset.apps/fluentd-elasticsearch   4         4         4       4            4           mylable=allow_demon_set   5m28s   fluentd-elasticsearch   quay.io/fluentd_elasticsearch/fluentd:v2.5.2   name=fluentd-elasticsearch

Kindly confirm if this what you want to do and works with kops

-- DT.
Source: StackOverflow