Using Digital Ocean Kubernetes Auto-Scaling for auto-downgrading node availability

5/5/2020

I have a Kubernetes cluster i.e "Cluster1" with auto-scaling to a maximum of 2 nodes & a minimum of 1 node. I am trying to understand the digital ocean behavior for auto-downgrading the nodes in the following scenario.

  1. The "Cluster1" cluster has an Nginx as the ingress-controller, which was added as part of the "1-click setup" during cluster provisioning.
  2. This cluster has auto-scaling configured as 1 min node & 2 max nodes. Let's call them Node1 & Node2.
  3. This cluster is behind a digital ocean load balancer i.e LB1 which talks to ingress controller i.e pod running Nginx.
  4. Let's say there is a single replica (replica:1) deployment controller of "image1" which requires 80% of the CPU.
  5. Initially, the image1 is deployed & since there is resource availability, the image1 starts running on Node1.
  6. Consider the image1 is updated to image2, upstream. The deployment controller will see there's node unavailability & will provision Node2, & will create another pod running image2 on Node2, the pod running image1 will start to terminate once the image2 is up and running.
  7. LB1 updates the routing to Node1, Node2.
  8. Now after the pod (on Node1) for image1 is terminated because the replica:1 is set in the deployment controller, the Node1 is not running anything from the user perspective.
  9. Ideally, there should be an automatic de-provisioning of the node i.e Node1.
  10. I tried to manually remove the Node1 from the cluster using the DO dashboard.
  11. LB1 updates & shows single node availability, but shows the status as down.
  12. Upon investigating I found that "nginx-controller" was running only on Node1. When the Node1 is terminated, the "nginx-controller" takes a while to provision a new pod on then available Node2. There is however downtime all this while.

My question is how-to best use auto-scaling for downgrading. I have a few solutions I thought.

  1. Is it possible to run "nginx-controller" on all nodes?

or

  1. If I drain the node from the "kubectl", i.e kubectl drain , and then delete manually delete the node from the dashboard, there shouldn't any downtime?. or just doing kubectl drain, will make DigitalOcean auto downgrade.
-- user1064504
digital-ocean
kubernetes
nginx

1 Answer

5/6/2020

if Nginx ingress controller is set as DaemonSet, it is possible

Please go through kubernetes official document about this topic.

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

-- BMW
Source: StackOverflow