Error: Status 403 trying to pull repository. kubernetes 1.5.4 specific

3/28/2017

I have two GCE clusters, they have identical configuration, only difference is kubernetes node version: 1.4.7 vs 1.5.4. Deployment on the first cluster successfully starts, however it fails on the second with status 403.

Here is the output of describe 1.5.4 pod:

Name:       essence-elasticsearch-2766004759-lr5wf
Namespace:  default
Node:       gke-nts-elastic-test-default-pool-115230a1-p17s/10.128.0.2
Start Time: Tue, 28 Mar 2017 16:25:00 +0300
Labels:     app=essence-elasticsearch-app
        pod-template-hash=2766004759
Status:     Pending
IP:     10.104.0.9
Controllers:    ReplicaSet/essence-elasticsearch-2766004759
Containers:
  essence-elasticsearch:
    Container ID:   
    Image:      gcr.io/en-development/essence-elasticsearch:f7bc95c_2e7df37_50683bc_d6e8032
    Image ID:       
    Port:       9200/TCP
    Requests:
      cpu:      100m
    State:      Waiting
      Reason:       ImagePullBackOff
    Ready:      False
    Restart Count:  0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-s0rp8 (ro)
    Environment Variables:  <none>
Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  default-token-s0rp8:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-s0rp8
QoS Class:  Burstable
Tolerations:    <none>
Events:
  FirstSeen LastSeen    Count   From                                SubObjectPath               Type        Reason      Message
  --------- --------    -----   ----                                -------------               --------    ------      -------
  14m       14m     1   {default-scheduler }                                            Normal      Scheduled   Successfully assigned essence-elasticsearch-2766004759-lr5wf to gke-nts-elastic-test-default-pool-115230a1-p17s
  14m       3m      7   {kubelet gke-nts-elastic-test-default-pool-115230a1-p17s}   spec.containers{essence-elasticsearch}  Normal      Pulling     pulling image "gcr.io/en-development/essence-elasticsearch:f7bc95c_2e7df37_50683bc_d6e8032"
  14m       3m      7   {kubelet gke-nts-elastic-test-default-pool-115230a1-p17s}   spec.containers{essence-elasticsearch}  Warning     Failed      Failed to pull image "gcr.io/en-development/essence-elasticsearch:f7bc95c_2e7df37_50683bc_d6e8032": Error: Status 403 trying to pull repository en-development/essence-elasticsearch: "Unable to access the repository: en-development/essence-elasticsearch; please verify that it exists and you have permission to access it."
  14m       3m      7   {kubelet gke-nts-elastic-test-default-pool-115230a1-p17s}                       Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "essence-elasticsearch" with ErrImagePull: "Error: Status 403 trying to pull repository en-development/essence-elasticsearch: \"Unable to access the repository: en-development/essence-elasticsearch; please verify that it exists and you have permission to access it.\""

  14m   8s  59  {kubelet gke-nts-elastic-test-default-pool-115230a1-p17s}   spec.containers{essence-elasticsearch}  Normal  BackOff     Back-off pulling image "gcr.io/en-development/essence-elasticsearch:f7bc95c_2e7df37_50683bc_d6e8032"
  14m   8s  59  {kubelet gke-nts-elastic-test-default-pool-115230a1-p17s}                       Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "essence-elasticsearch" with ImagePullBackOff: "Back-off pulling image \"gcr.io/en-development/essence-elasticsearch:f7bc95c_2e7df37_50683bc_d6e8032\""
-- vglazkov
google-cloud-platform
google-kubernetes-engine
kubernetes

1 Answer

3/28/2017

Are the two clusters in the same project? It looks like the 2nd one can't pull gcr.io/en-development/essence-elasticsearch

-- Janos Lenart
Source: StackOverflow