How to fix ephemeral local storage problem?

5/4/2020

I'm Running some deployment on EKS k8s 1.16 and after ~5 min my pod gets Evicted with the following message:

Pod ephemeral local storage usage exceeds the total limit of containers 1Gi.

My node has 20Gi ephemeral storage.

My QoS Class is Guaranteed and no matter which amount of ephemeral-storage I configure in my yaml, I see the same error with the amount I configure.

Do you have a clue what can be done?

My yaml file is here: https://slexy.org/view/s2096sex7L

-- alexarsh
amazon-eks
kubernetes

1 Answer

5/4/2020

It's because you're putting an upper limit of ephemeral-storage usage by setting resources.limits.ephemeral-storage to 1Gi. Remove the limits.ephemeral-storage if safe, or change the value depending upon your requirement.

      resources:
        limits:
          memory: "61Gi"
          cpu: "7500m"
          ephemeral-storage: "1Gi" <----- here
        requests:
          memory: "61Gi"
          cpu: "7500m"
          ephemeral-storage: "1Gi"

If the node where a Pod is running has enough of a resource available, it’s possible (and allowed) for a container to use more resources than its request for that resource specifies. However, a container is not allowed to use more than its resource limit.

-- Kamol Hasan
Source: StackOverflow