K
Q

What happens when one of the EKS worker nodes runs out of allocated EBS storage?

2/2/2021

I've created an EKS cluster using this terraform module:

module "eks" {
  ...
  worker_groups = [
    {
      name                          = "worker-group-1"
      instance_type                 = "t2.small"
      asg_desired_capacity          = 2
    }
  ]
}

As you can see, It creates a worker group of t2.small EC2 instances. t2.small instances are EBS-backed and have 20 GiB of EBS volume by default. What happens when one of these nodes, consumes all of it's allocated EBS volume?

  • Does the cluster autoscaler (which is enabled in my EKS cluster) create a new worker node?
  • Or, Does the allocated EBS volume get increased?

If none of the above scenarios happens, How should I deal with it in my cluster? What's the best approach?

-- HsnVahedi
amazon-ebs
amazon-ec2
amazon-eks
kubernetes

Similar Questions

How to configure Jenkins master and dynamic agents in different clouds? With Kubernetes plugin
How to expose a Kubernetes service on a specific Nodeport?
Pulumi: retrieve kubernetes secret value
Kubernetes's Ingress has no effect on nginx-ingress-controller
nginx-ingress doesn't work with AWS ELB when use aws-load-balancer-backend-protocol: "https"

1 Answer

2/2/2021

What happens when one of these nodes, consumes all of it's allocated EBS volume?

  • The node reports a condition when a compute resource is under pressure. The scheduler views that condition as a signal to dissuade placing additional pods on the node. In this case, the Node Condition would be DiskPressure which means that no new Pods are scheduled to the node. You can find more details regarding that topic by checking Configure Out of Resource Handling docs.

Does the cluster autoscaler (which is enabled in my EKS cluster) create a new worker node? Or, Does the allocated EBS volume get increased?

  • The autoscaler checks the cluster for pods that cannot be scheduled on any existing nodes because of inadequate CPU or memory resources or because the pod’s node affinity rules or taint tolerations do not match an existing node. If the cluster has unschedulable pods, the autoscaler will check its managed node pools to decide if adding a node would unblock the pod. If so, it will add a node if the node pool can be increased in size. All the specifics of AWS implementation of the Cluster Autoscaler can be found here.
-- Wytrzymały Wiktor
Source: StackOverflow