I am looking for some suggestions on debugging and root causing an issue on one of my nodes in a Kubernetes cluster.

The issue is any docker pull operation on that node fails with a 'not found' error.

here is what I have tried so far.

1. Checked that the private registry is ok and works from other nodes.
2. Checked docker daemon is running, other docker operations work
3. No issues with networking (dig works, curl -v https://registry.abc.xyz.com.symcpe.net:443https:// --> returns HTTP/1.1 200 OK)

I have not tried restarting the docker daemon, frankly I am scared to do it since there could be production pods on this node.

Any docker pull operation fails on that node with a 'not found' error. It is not an issue with the registry as the operation works from other nodes.

Tried both with our private registry and docker.io

# docker -D pull  registry.abc.xyz.com/behzad_dastur/brokermon:0.1
Trying to pull repository registry.abc.xyz.com/brd/brokermon 
... 0.1: Pulling from brd/brokermon
not found
Error: image brd/brokermon:0.1 not found

# docker pull alpine:latest
Trying to pull repository docker.io/library/alpine ... latest: Pulling from 
library/alpine
not found
Tag latest not found in repository docker.io/library/alpine

Docker version:

# docker version
Client:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-25.1.origin.el7.x86_64
 Go version:      go1.4.2
 Git commit:      78ee77d/1.9.1
 Built:           
 OS/Arch:         linux/amd64

Server:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-25.1.origin.el7.x86_64
 Go version:      go1.4.2
 Git commit:      78ee77d/1.9.1
 Built:           
 OS/Arch:         linux/amd64

Docker search output:

# docker search alpine
INDEX       NAME                                             DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
docker.io   docker.io/alpine                                 A minimal Docker image based on Alpine Lin...   2818      [OK]       
docker.io   docker.io/mhart/alpine-node                      Minimal Node.js built on Alpine Linux           323                  
docker.io   docker.io/frolvlad/alpine-oraclejdk8             The smallest Docker image with OracleJDK 8...   269                  [OK]
docker.io   docker.io/anapsix/alpine-java                    Oracle Java 8 (and 7) with GLIBC 2.23 over...   253                  [OK]
docker.io   docker.io/gliderlabs/alpine                      Image based on Alpine Linux will help you ...   171