After a few days of running dockerd on a kubernetes host, where pods are scheduled by kubelet, dockerd goes bad - consuming a lot of resources (50% memory - ~4gigs).

When it gets to this state, it is unable to act on commands for containers that appear to be running via $ docker ps. Also checking ps -ef on the host these containers don't map to any underlying host processes.

$ docker exec yields -

level=error msg="Error running exec in container: rpc error: code = 2 desc = containerd: container not found"

Cannot kill container 6a8d4....8: rpc error: code = 14 desc = grpc: the connection is unavailable"

level=fatal msg="open /var/run/docker/libcontainerd/containerd/7657...4/65...6/process.json: no such file or directory"

Looking through the process tree on the host there seem to be a lot of defunct processes which point to dockerd as the parent id. Any pointers on what the issue might be or where to look further?

Enabled debug on dockerd to see if the issue re-occurs, a dockerd restart fixes the issue.