This seems to be an issue with hard link limits in coreos (?) that should have been resolved with Docker overlayfs2. We don't know why this is occurring in kubernetes (1.8) however since it should be automatically garbage collecting old images.
Any ideas?
Error from kubectl describe:
Warning Failed 2m kubelet, gke-dev-ctp-default-pool-08c10bb8-f2d1
Failed to pull image "gcr.io/my-group/my-image:latest": rpc error: code = Unknown desc = failed to register layer: link /var/lib/docker/overlay/2f8076d1832a8a60086988b73fd2d8eec7ba38e62e9615f592d048d72d48a227/root/usr/libexec/git-core/git-merge-tree /var/lib/docker/overlay/0c6044f911da4845bac37bad4d1fcf9a7e03a7b8242cf2bf8da46e17ec23e3f9/tmproot518352995/usr/libexec/git-core/git-merge-tree: too many linksGKE is currently not using overlayfs2. They are working on enabling this feature in GKE version 1.9 along with the new docker version.
The workaround is to remove all the unused images manually using the following method:
$ docker images | wc -l
$ docker image prune -a
$ docker images | wc -l