My service running in a pod output too much log and cause low ephemeral storage. As a result, the pod is evicted and other services can't deploy to k8s.
So how I can determine pod resource ephemeral storage requests and limit to avoid this situation? I can't find any best practice about ephemeral storage.
Note that by default, if you have not set any limits on ephemeral-storage the pod has access to the entire disk of the node it is running on, so if you are certain that the pod is being evicted because of this, then you are certain that the pod consumed it all. You can check this from kubelet logs, as kubelet is the guy in charge of detecting this behavior and evicting the pod.
Now, here you have two options. Either you can set an ephemeral-storage limit, and make a controlled pod eviction, or just get an external volume, map it into the container, and get the logs outside of the node.
You can also monitor the disk usage, as suggesting shubham_asati, but if it is eating it all, it is eating it all. You are just going to look at how it is getting filled out.
I guess ephemeral storage for a pod can be defined as cpu request/limit. See this https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#local-ephemeral-storage but this feature is in beta stage K8's version 1.16.
To check namespace level resource consumption view https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota.
You can set request/limit ephemeral storage for each pod .
Regarding your issue
then compare the storages from both outputs.
You need to deploy prometheus and grafana to find out how much memory and cpu are getting consumed by the pod. Then accordingly set those request and limits on that pod
Requests and limits setting for ephemeral storage is a new feature and is still in beta.You might have to wait few more months to use that feature.
However, if you are on k8s 1.18 then you can test Requests and limits setting for ephemeral storage