I have a GKE cluster which has been running fine up until recently. Now I see a whole bunch of Kubernetes Workloads showing as offline with the following error msg:

 Type     Reason          Age                    From                                                          Message
  ----     ------          ----                   ----                                                          -------
  Normal   Scheduled       6m23s                  default-scheduler                                               Warning  Failed          5m39s (x3 over 6m22s)  kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Error: ErrImagePull
  Normal   BackOff         5m2s (x7 over 6m21s)   kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Back-off pulling image "us.gcr.io/project/poc-app-bot@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8"
  Normal   Pulling         4m49s (x4 over 6m22s)  kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  pulling image "us.gcr.io/project/poc-app-bot@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8"
  Warning  Failed          81s (x22 over 6m21s)   kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Error: ImagePullBackOff

Not sure what could have changed to cause this issue.

This is the ouput of kubectl

Name:               project-5dddbd66b5-vpw8q
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-platsol-bots-staging-default-pool-f489f2f3-rjrq/10.x.x.x
Start Time:         Wed, 18 Sep 2019 16:48:23 +0100
Labels:             app=bot
                    pod-template-hash=5dddbd66b5
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container project
Status:             Pending
IP:                 10.20.1.9
Controlled By:      ReplicaSet/bot-5dddbd66b5
Containers:
  project:
    Container ID:
    Image:          us.gcr.io/project/project@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8
    Image ID:
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-99cns:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-99cns
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason  Age                     From                                                          Message
  ----     ------  ----                    ----                                                          -------
  Warning  Failed  4m38s (x793 over 3h4m)  kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Error: ImagePullBackOff

Below is what i have in my YAML definition for the deployment. I have not defined a secret as one was not required to pull the image from Google Container Registry,

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "3"
        kubectl.kubernetes.io/last-applied-configuration: |
<redacted annotations>
      creationTimestamp: 2019-06-06T08:37:01Z
      generation: 3
      labels:
        app: project
      name: bot
      namespace: default
      resourceVersion: "68945490"
      selfLink: /apis/apps/v1/namespaces/default/deployments/bot
      uid: 412ce711-8836-11e9-905f-42010a8e016c
     image: us.gcr.io/project/app-bot@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8
        imagePullPolicy: IfNotPresent

Okay so I followed this guide to patch the service account with a "secret" when pulling images from GCR https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

I SSH onto a single node and can pull an image for one Application successfully,

vinay@cloudshell:~ (project-id)$ docker pull us.gcr.io/project-id/project2-bot@sha256:9817462c743a93bb9206e4b8685
5322f731a768dca18e26b8bfc39b0cc886d31
sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8bfc39b0cc886d31: Pulling from project-id/project2-bot
092586df9206: Pull complete
ef599477fae0: Pull complete
4530c6472b5d: Pull complete
d34d61487075: Pull complete
272f46008219: Pull complete
12ff6ccfe7a6: Pull complete
f26b99e1adb1: Pull complete
bb50901cd579: Pull complete
64a286652062: Pull complete
283785ced197: Pull complete
ed5a2062edd6: Pull complete
Digest: sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8bfc39b0cc886d31
Status: Downloaded newer image for us.gcr.io/project-id/project2-bot@sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8
bfc39b0cc886d31
us.gcr.io/project-id/project2-bot@sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8bfc39b0cc886d31

But this application seems to throw an error,

vinay@cloudshell:~ (project-id)$ docker pull us.gcr.io/project-id/project1-plug@sha256:c53ac1c536a1187ce940f9221730cc0eae3103f4313033659e2162a70bc66c59
    sha256:c53ac1c536a1187ce940f9221730cc0eae3103f4313033659e2162a70bc66c59: Pulling from project-id/project1-plug
    a4d8138d0f6b: Pulling fs layer
    dbdc36973392: Pulling fs layer
    f59d6d019dd5: Pulling fs layer
    aaef3e026258: Waiting
    5e86b04a4500: Waiting
    1a6643a2873a: Waiting
    2ad1e30fc17c: Waiting
    ddb5baaf3393: Waiting
    0a7edc889b3c: Waiting
    31a1f16c256b: Waiting
    172a500f7b4d: Waiting
    error pulling image configuration: unknown blob