Kubernetes Garbage Collection fails - FreeDiskSpaceFailed & ImageGCFailed

8/1/2017

Apparently the GC of my Kubernetes cluster is failing to delete any image and the server is getting to full-disk.

Can you please guide me on where to find the logs for the ImageGC with the error trying to delete the images or to a reason of why this is happening?

3m         5d          1591      ip-xxx.internal     Node                                          Warning   FreeDiskSpaceFailed       {kubelet ip-xxx.internal}     failed to garbage collect required amount of images. Wanted to free 6312950988, but freed 0
3m         5d          1591      ip-xxx.internal     Node                                          Warning   ImageGCFailed             {kubelet ip-xxx.internal}     failed to garbage collect required amount of images. Wanted to free 6312950988, but freed 0

Thanks!

-- felipeclopes
docker
google-kubernetes-engine
kubernetes

1 Answer

8/2/2017

most likely your host file system is full, can you check /var file system usage. you can use docker-gc to cleanup old image.

https://github.com/spotify/docker-gc

Run it like this

docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc
-- sfgroups
Source: StackOverflow