I tried to set up my own container on GKE using gcr.io and keep getting ImagePullBackOff failure.

Thinking I was doing something wrong, I went back to the tutorial here https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app and followed all the steps and get the same error. It looks like a credential problem but I follow all the steps of the tutorial and still no luck.

How do I debug this error as the logs don't seem to help.

steps 1-4 of the tutorial work.

kubectl run hello-web --image=gcr.io/${PROJECT_ID}/hello-app:v1 --port 8080

fails with ImagePullBackOff I thought GKE and gcr.io handle credentials automatically. What am I doing wrong? How do I debug this?

kubectl describe pods hello-web-6444d588b7-tqgdm

Name:           hello-web-6444d588b7-tqgdm
Namespace:      default
Node:           gke-aia-default-pool-9ad6a2ee-j5g7/10.152.0.2
Start Time:     Sat, 27 Oct 2018 06:51:38 +1000
Labels:         pod-template-hash=2000814463
                run=hello-web
Annotations:    kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container hello-web
Status:         Pending
IP:             10.12.2.5
Controlled By:  ReplicaSet/hello-web-6444d588b7
Containers:
hello-web:
    Container ID:   
    Image:          gcr.io/<project-id>/hello-app:v1
    Image ID:       
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
    Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
    cpu:        100m
    Environment:  <none>
    Mounts:
    /var/run/secrets/kubernetes.io/serviceaccount from default-token-qgv8h (ro)
Conditions:
Type           Status
Initialized    True 
Ready          False 
PodScheduled   True 
Volumes:
default-token-qgv8h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qgv8h
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type     Reason                 Age                  From                                         Message
----     ------                 ----                 ----                                         -------
Normal   Scheduled              45m                  default-scheduler                            Successfully assigned hello-web-6444d588b7-tqgdm to gke-aia-default-pool-9ad6a2ee-j5g7
Normal   SuccessfulMountVolume  45m                  kubelet, gke-aia-default-pool-9ad6a2ee-j5g7  MountVolume.SetUp succeeded for volume "default-token-qgv8h"
Normal   Pulling                44m (x4 over 45m)    kubelet, gke-aia-default-pool-9ad6a2ee-j5g7  pulling image "gcr.io/<project-id>/hello-app:v1"
Warning  Failed                 44m (x4 over 45m)    kubelet, gke-aia-default-pool-9ad6a2ee-j5g7  Failed to pull image "gcr.io/<project-id>/hello-app:v1": rpc error: code = Unknown desc = Error response from daemon: repository gcr.io/<project-id>/hello-app not found: does not exist or no pull access
Warning  Failed                 44m (x4 over 45m)    kubelet, gke-aia-default-pool-9ad6a2ee-j5g7  Error: ErrImagePull
Normal   BackOff                5m (x168 over 45m)   kubelet, gke-aia-default-pool-9ad6a2ee-j5g7  Back-off pulling image "gcr.io/<project-id>/hello-app:v1"
Warning  Failed                 48s (x189 over 45m)  kubelet, gke-aia-default-pool-9ad6a2ee-j5g7  Error: ImagePullBackOff

cluster permissions:

User info Disabled
Compute Engine Read/Write
Storage Read Only
Task queue Disabled
BigQuery Disabled
Cloud SQL Disabled
Cloud Datastore Disabled
Stackdriver Logging API Write Only
Stackdriver Monitoring API Full
Cloud Platform Disabled
Bigtable Data Disabled
Bigtable Admin Disabled
Cloud Pub/Sub Disabled
Service Control Enabled
Service Management Read Only
Stackdriver Trace Write Only
Cloud Source Repositories Disabled
Cloud Debugger Disabled