Currently running a k8s cluster however occasionally I get memory issues. The following error will pop up,
Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "<web app>": Error response from daemon: devmapper: Thin Pool has 6500 free data blocks which is less than minimum required 7781 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
I can resolve this by manually running docker ps -a -f status=exited -q | xargs -r docker rm -v
However I want Kubernetes to do this work itself. Currently in my kublet config I have:
evictionHard:
imagefs.available: 15%
memory.available: "100Mi"
nodefs.available: 10%
nodefs.inodesFree: 5%
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
What am i doing wrong?
Reading the error you've posted seems to me you are using "devicemapper" as storage driver.
The
devicemapper
storage driver is deprecated in Docker Engine 18.09, and will be removed in a future release. It is recommended that users of thedevicemapper
storage driver migrate tooverlay2
.
I should suggest you use "overlay2" as storage drive, unless you are running a non-support OS. See here the support OS versions.
You can check your actual storage drive using docker info
command, you will get an output like this:
Client:
Debug Mode: false
Server:
Containers: 21
Running: 18
Paused: 0
Stopped: 3
Images: 11
Server Version: 19.03.5
Storage Driver: devicemapper <<== See here
Pool Name: docker-8:1-7999625-pool
Pool Blocksize: 65.54kB
...
> Supposing you want to change the storage drive from devicemapper
to overlay2
, you need to following this steps:
Changing the storage driver makes existing containers and images inaccessible on the local system. Use docker save
to save any images you have built or push them to Docker Hub or a private registry before changing the storage driver, so that you do not need to re-create them later.
Before following this procedure, you must first meet all the prerequisites.
Stop Docker.
$ sudo systemctl stop docker
Copy the contents of /var/lib/docker
to a temporary location.
$ cp -au /var/lib/docker /var/lib/docker.bk
If you want to use a separate backing filesystem from the one used by /var/lib/
, format the filesystem and mount it into /var/lib/docker
. Make sure add this mount to /etc/fstab
to make it permanent.
Edit /etc/docker/daemon.json
. If it does not yet exist, create it. Assuming that the file was empty, add the following contents.
{
"storage-driver": "overlay2"
}
Docker does not start if the daemon.json
file contains badly-formed JSON.
Start Docker.
$ sudo systemctl start docker
Verify that the daemon is using the overlay2
storage driver. Use the docker info
command and look for Storage Driver
and Backing filesystem
.
Client:
Debug Mode: false
Server:
Containers: 35
Running: 15
Paused: 0
Stopped: 20
Images: 11
Server Version: 19.03.5
Storage Driver: overlay2 <=== HERE
Backing Filesystem: extfs <== HERE
Supports d_type: true
Extracted from Docker Documentation.