K
Q

AWS EKS Node - Volume Attach Limit

6/6/2019

We recently upgraded our EKS environment to v1.12.7. After the upgrade we noticed that there is now an "allocatable" resource called attachable-volumes-aws-ebs and in our environment we have many EBS volumes attached to each node (they were all generated dynamically via PVCs).

Yet on every node in the "allocated resources" section, it shows 0 volumes attached:

Allocatable:
 attachable-volumes-aws-ebs:  25
 cpu:                         16
 ephemeral-storage:           96625420948
 hugepages-1Gi:               0
 hugepages-2Mi:               0
 memory:                      64358968Ki
 pods:                        234
...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests           Limits
  --------                    --------           ------
  cpu                         5430m (33%)        5200m (32%)
  memory                      19208241152 (29%)  21358Mi (33%)
  attachable-volumes-aws-ebs  0                  0

Because of this, the scheduler is continuing to try and attach new volumes to nodes that already have 25 volumes attached.

How do we get kubernetes to recognize the volumes that are attached so that the scheduler can act accordingly?

-- viperseph
amazon-eks
aws-eks
kubernetes

Similar Questions

Does Kubernetes consider the current memory usage when scheduling pods
Kubernetes Jobs not Completing
User "system:anonymous" cannot create resource "pods" in API group "" in the namespace "default"
terraform (0.11 and 0.12) apply works on 1 machine, but not on the other
setup elasticsearch cluster on aks azure using terraform
Is it possible to make a FROM instruction in a Dockerfile pull the most recent image?
How do I run a Docker pre-entrypoint command without knowing the original entrypoint?
How to access minikube IP and service port in container?

1 Answer

6/19/2019

First check your pods status, they can have pending status that's why probably you volumes does't count. Your volume could stuck in attaching state when using multiple PersistentVolumeClaim too.

Your volumes may be not attached due to NodeWithImpairedVolumes=true:NoSchedule flag and at the same time number of your attachable-volumes-aws-ebs.

Try to execute:

$ kubectl taint nodes node1 key:NoSchedule-

on every node with this label (NodeWithImpairedVolumes=true:NoSchedule).

If you use awsElasticBlockStore , there are some restrictions when using an awsElasticBlockStore volume:

  • nodes on which Pods are running must be AWS EC2 instances those instances need to be in the same region and availability-zone as the EBS volume EBS only supports a single EC2 instance mounting a volume.

You can use count/* resource quota, an object is charged against the quota if it exists in server storage. These types of quotas are useful to protect against exhaustion of storage resources use count/persistentvolumeclaims.

Allocatable will be computed by the Kubelet and reported to the API server. It is defined to be:

[Allocatable] = [Node Capacity] - [Kube-Reserved] - [System-Reserved] - [Hard-Eviction-Threshold]

Note: Since kernel usage can fluctuate and is out of kubernetes control, it will be reported as a separate value (probably via the metrics API). Reporting kernel usage is out-of-scope for this proposal.

-- MaggieO
Source: StackOverflow