Container image exists in registry but pod shows 'ErrImagePull'

7/30/2018

I'm trying to deploy an image into IBM Cloud kubernetes instance.

First I created and pushed an image ...

$ ic cr images
Listing images...

REPOSITORY                                    NAMESPACE     TAG      DIGEST         CREATED          SIZE     VULNERABILITY STATUS   
registry.ng.bluemix.net/my_namespace/my_app   my_namespace  latest   123456789012   56 minutes ago   264 MB   Vulnerable   

OK

Then ...

$ ic cs cluster-config my_cluster
$ export KUBECONFIG=/Users/me/.bluemix/plugins/container-service/clusters/my_cluster/kube-config-mil01-my_cluster.yml

$ kubectl run my_app --image=registry.ng.bluemix.net/my_namespace/my_app \
          --port=8080 --generator=run/v1
replicationcontroller/my_app created

And ...

$ kubectl get pods
NAME          READY     STATUS         RESTARTS   AGE
my_app-zvqwq  0/1       ErrImagePull   0          38s

Although my region is us-south ...

$ ic target

API endpoint:      https://api.ng.bluemix.net   
Region:            us-south   
User:              me@email.com   

... interestingly the cluster was created in milan ...

$ ic cs clusters
OK
Name          ID               State    Created          Workers   Location   Version   
my_cluster    xxxxxxxxxxxxxx   normal   55 minutes ago   1         mil01      1.9.9_1520 

Update Thanks to Bloodysock here is some more debug info:

Failed to pull image "registry.ng.bluemix.net/my_namespace/my_app": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.ng.bluemix.net/v2/my_namespace/my_app/manifests/latest: unauthorized: authentication required

However, I can't see in the docs how to setup authentication for the IBM Container Registry.

-- Chris Snow
ibm-cloud
kubernetes

1 Answer

7/30/2018

The message unauthorized: authentication required suggests that your pod doesn't have a valid ImagePullSecret for the US South (ng) registry. Since your cluster was created in Milan, the secret that is installed by IKS when you create your cluster may well be for the EU Central (eu-de) registry, rather than US South.

If you have the ability to view secrets in your cluster, you can check the contents of the automatically installed ImagePullSecret by running kubectl get secret bluemix-default-secret-regional -o yaml | grep .dockercfg: | awk '{print $2}' | base64 --decode. This command gets the secret called bluemix-default-secret-regional, extracts the .dockercfg field, and then decodes the base64 encoded secret value.

You can create additional ImagePullSecrets and optionally add them to your default ServiceAccount in order for them to be automatically used by your cluster. For instructions, see here.

By the way, your cluster might have been created in Milan if your ic cs plugin is targeting the EU Central region for IKS. You can view what IKS region you're using with ic cs region, and change it with ic cs region-set.

-- molepigeon
Source: StackOverflow