kubernetes is un-able to pull large image from private repository in docker-hub

9/5/2018

lately I started using private repositories in docker-hub with my kubernetes cluster.

my kubernetes cluster version is 1.9.2 ( kubelet , kubeadm , kubectl )

my kubernetes cluster has about 12 deployments and they are all in default namespace.

I also created a docker registry secret:

kubectl create secret docker-registry registrykey ..

and I patched it into default namespace:

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "registrykey"}]}' 

all my deployments works perfectly except one which keeps giving me ImagePullBackOff

when I try to describe it I get:

 Normal   BackOff  29m (x6112 over 1d)  kubelet, node1  Back-off pulling image "registry.hub.docker.com/my-image"
  Warning  Failed   19m (x6155 over 1d)  kubelet, node1  Error: ImagePullBackOff
  Warning  Failed   14m (x287 over 1d)   kubelet, node1  Failed to pull image "registry.hub.docker.com/my-image": rpc error: code = Unknown desc = Error response from daemon: repository registry.hub.docker.com/my-image not found: does not exist or no pull access
  Normal   Pulling  3m (x289 over 1d)    kubelet, node1  pulling image "registry.hub.docker.com/my-image"

I tried adding to kubelet service the following flags:

--image-pull-progress-deadline=5m --serialize-image-pulls=false

I also added docker service the following flag: --max-concurrent-downloads=10

the pull still fails.

the size of the image is 1.56 giga.

if I pull it manually via docker pull and change imagePullPolicy: Always to imagePullPolicy: IfNotPresent, it works. since I don't want to do it manually every deployment I am stuck.

any idea why it is happening and how to solve it will be welcomed.

many thanks

edit: added service account yaml:

apiVersion: v1
imagePullSecrets:
- name: registrykey
kind: ServiceAccount
metadata:
  creationTimestamp: 2018-09-04T06:55:04Z
  name: default
  namespace: default
  resourceVersion: "11698"
  selfLink: /api/v1/namespaces/default/serviceaccounts/default
  uid: 73c55e52-b00f-11e8-afec-0050569e63f0
secrets:
- name: default-token-ptx9l

and secret:

apiVersion: v1
data:
  .dockerconfigjson: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{".dockerconfigjson":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"},"kind":"Secret","metadata":{"annotations":{},"creationTimestamp":null,"name":"registrykey","namespace":"default"},"type":"kubernetes.io/dockerconfigjson"}
  creationTimestamp: 2018-09-04T08:41:40Z
  name: registrykey
  namespace: default
  resourceVersion: "11689"
  selfLink: /api/v1/namespaces/default/secrets/registrykey
  uid: 57ebf775-b01e-11e8-a16b-0050569e63f0
type: kubernetes.io/dockerconfigjson

I am using a docker hub private repositories

I am still desperate. has any one ever encountered this behavior before?

-- eran meiri
docker
docker-registry
kubernetes

0 Answers