failed to garbage collect required amount of images. Wanted to free 6283487641 bytes, but freed 0 bytes

3/31/2021

I have searched many websites and articles but not found any perfect answer. I am using eks version 1.18. I can see a few of the pods are "Evicted", but when trying to check the node I can see the error "(combined from similar events): failed to garbage collect required amount of images. Wanted to free 6283487641 bytes, but freed 0 bytes".

Is there any way we can find the reason why it's failing? or how to fix this issue? Any suggestions are most welcome.

enter image description here

I can see the disk "overlays" filesystem is almost full within a few hours. I am not sure what's going on. The below screenshot shows my memory utilization.

enter image description here

-- JDGuide
amazon-ec2
amazon-eks
eksctl
kubernetes
nodes

1 Answer

4/12/2021

see if you can change the Kubernetes GC policies. I guess the issues may be due to recent changes in the flags

the new ones are using the flags as --eviction syntax, can you check if that is the case with your setup causing the failure on clearing the space

Please refer to the docs here

https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/

-- NBaua
Source: StackOverflow