pulling image from private docker registry causes filesystem layer verification failed for digest sha256:

9/30/2021

So I have a Kubernetes cluster which is connected to a private docker registry. One of my nodes in Kubernetes keeps failing at pulling an image. So I went into the node and tried to pull the image manually. When I try to pull the image, it fails. I get the following error: filesystem layer verification failed for digest sha256:...

I tried the following post, but no solution in there worked: https://stackoverflow.com/questions/34448975/filesystem-layer-verification-failed-for-digest

-- ARR
docker
docker-registry
kubernetes

1 Answer

9/30/2021

I have a private docker registry, a layer was failing for verification right after it was downloaded, none of the above solutions worked for me.

I did the following, delete the image and specific layer from the private docker registry. Restart the private docker registry. Then rebuild and push the image again. It should work afterwards.

To remove a tag from a private docker registry I use the following script (Python):

image = "yourimage"
tag = "yourtag"
host = "https://yourhost:5000" # change to http if you have no ssl
username = "yourusername"
password = "yourpassword"
res = requests.get("{}/v2/{}/manifests/{}".format(host, image, tag), auth=HTTPBasicAuth(username, password), verify=False, headers={
        "Accept": "application/vnd.docker.distribution.manifest.v2+json"
    })
digest = res.headers.get("Docker-Content-Digest").replace('"', "")
res = requests.delete("{}/v2/{}/manifests/{}".format(host, image, digest), auth=HTTPBasicAuth(username, password), verify=False)
    print(res.status_code)

to remove the blob, your blob digest is the sha256:.... that fails at verification

image = "yourimage"
host = "https://yourhost:5000" # change to http if you have no ssl
username = "yourusername"
password = "yourpassword"
blob_digest = "yourblobdigest"
res = requests.delete("{}/v2/{}/blobs/{}".format(host, image, blob_digest), auth=HTTPBasicAuth(username, password),
                      verify=False)

after you have finished this go to your container of your private docker registry and remove the image like this:

rm /var/lib/registry/docker/registry/v2/repositories/your-repository-name

Exit your docker registry and restart your private docker registry container. Build and push the image and it should all go fine when when you pull again.

-- ARR
Source: StackOverflow