I specified ephemeral storage limits and requests on a container, and I’m using latest amazon-eks-node-1.12-v20190701 image.
My issue is that when I specify ephemeral-storage: 1Mi my pod is being evicted for using too much. I can easily add more but I would like to know how exactly this works and kubernetes documentation is not very helpful.
From what I read, there is a Garbage Collection that will clean up unused images and unused containers. Kubelet will perform garbage collection for containers every minute and garbage collection for images every five minutes.
Is that apply to ephemeral storage too?
How do I check what’s inside the ephemeral storage (via container or a node?) and how can I reduce it?
Is it true that when ephemeral-storage is up to 85% and when garbage collection run, it will be cleaned up?
Sorry for a lot of questions, just trying to understand and it’s difficult to find a simple answe
As far as I'm aware (please correct me if this is wrong) there is no Garbage Collection for Ephemeral Storage.
As we can read in the Kubernetes documentation regarding Garbage Collection.
Some Kubernetes objects are owners of other objects. For example, a ReplicaSet is the owner of a set of Pods. The owned objects are called dependents of the owner object. Every dependent object has a
metadata.ownerReferencesfield that points to the owning object.Sometimes, Kubernetes sets the value of
ownerReferenceautomatically. For example, when you create a ReplicaSet, Kubernetes automatically sets theownerReferencefield of each Pod in the ReplicaSet. In 1.8, Kubernetes automatically sets the value ofownerReferencefor objects created or adopted by ReplicationController, ReplicaSet, StatefulSet, DaemonSet, Deployment, Job and CronJob.You can also specify relationships between owners and dependents by manually setting the
ownerReferencefield.
So we can take advantage of Garbage Collection to delete dependent objects.
When you delete an object, you can specify whether the object’s dependents are also deleted automatically. Deleting dependents automatically is called cascading deletion. There are two modes of cascading deletion: background and foreground.
If you delete an object without deleting its dependents automatically, the dependents are said to be orphaned.
In Kubernetes v1.16 new resource was introduced Local ephemeral storage >
In each Kubernetes node, kubelet’s root directory (/var/lib/kubelet by default) and log directory (/var/log) are stored on the root partition of the node. This partition is also shared and consumed by Pods via emptyDir volumes, container logs, image layers and container writable layers.
And to answer your question
How do I check what’s inside the ephemeral storage (via container or a node?) and how can I reduce it?
It might be possible to exec into the docker container that have the storage mounted and see whats there, but other then that I don't see a way of accessing the content. As for reducing the storage (I did not tested that) it might remove the content of the ephemeral storage because it's not extendable.