In a pod, I have a different image specified that then one actually running (show in the statues), despite having an AlwaysPull policy. This seems like it should never happen. Is this a bug with GKE or something similar or is there actually a scenario where this is a valid state?

My requested image is the following:

image: gcr.io/basilica-211201/leadgenie-scraper-go:v0.0.3-9-g824c877

But this is the one in the container status:

image: gcr.io/basilica-211201/leadgenie-scraper-go:v0.0.3-8-gaa6cec6

Some of the other pods in the same ReplicaSet do have the correct image.

Not sure if there's any other information I could provide but here are definitions.

I have the following pod:

$ kubectl get pods scraper-go-router-7c564d75bd-n5rzl -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
      app'
  creationTimestamp: "2019-11-14T21:36:09Z"
  generateName: scraper-go-router-7c564d75bd-
  labels:
    app.kubernetes.io/name: scraper-go-router
    pod-template-hash: 7c564d75bd
  name: scraper-go-router-7c564d75bd-n5rzl
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: scraper-go-router-7c564d75bd
    uid: b647dc31-0726-11ea-b7df-42010a80006b
  resourceVersion: "57185702"
  selfLink: /api/v1/namespaces/default/pods/scraper-go-router-7c564d75bd-n5rzl
  uid: c5dcb19d-0726-11ea-b7df-42010a80006b
spec:
  containers:
  - command:
    - /root/router
    image: gcr.io/basilica-211201/leadgenie-scraper-go:v0.0.3-9-g824c877
    imagePullPolicy: Always
    name: app
    ports:
    - containerPort: 8080
      name: http
      protocol: TCP
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: gke-leadgenie-prod-nap-n1-highcpu-16--cf0692f1-6td6
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-11-14T21:36:09Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-11-14T21:36:40Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-11-14T21:36:40Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-11-14T21:36:09Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://7c0b86a2535b4a133796fcb544f2d6f5a1527fa588bad2f0866431fd3df29a7f
    image: gcr.io/basilica-211201/leadgenie-scraper-go:v0.0.3-8-gaa6cec6
    imageID: docker-pullable://gcr.io/basilica-211201/leadgenie-scraper-go@sha256:0415e1acfaea908d116392ea3ab5a3cd3e74495bb72fb1273d60c3d11888ec4c
    lastState: {}
    name: app
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2019-11-14T21:36:35Z"
  hostIP: 10.128.0.109
  phase: Running
  podIP: 10.48.118.251
  qosClass: Burstable
  startTime: "2019-11-14T21:36:09Z"

With the following replica set:

$ kubectl get rs scraper-go-router-7c564d75bd -o yaml
apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
  annotations:
    deployment.kubernetes.io/desired-replicas: "17"
    deployment.kubernetes.io/max-replicas: "22"
    deployment.kubernetes.io/revision: "16"
  creationTimestamp: "2019-11-14T21:35:43Z"
  generation: 1982
  labels:
    app.kubernetes.io/name: scraper-go-router
    pod-template-hash: 7c564d75bd
  name: scraper-go-router-7c564d75bd
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: Deployment
    name: scraper-go-router
    uid: 74441191-04f0-11ea-955f-42010a800206
  resourceVersion: "59007652"
  selfLink: /apis/extensions/v1beta1/namespaces/default/replicasets/scraper-go-router-7c564d75bd
  uid: b647dc31-0726-11ea-b7df-42010a80006b
spec:
  replicas: 17
  selector:
    matchLabels:
      app.kubernetes.io/name: scraper-go-router
      pod-template-hash: 7c564d75bd
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: scraper-go-router
        pod-template-hash: 7c564d75bd
    spec:
      containers:
      - command:
        - /root/router
        image: gcr.io/basilica-211201/leadgenie-scraper-go:v0.0.3-9-g824c877
        imagePullPolicy: Always
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 17
  fullyLabeledReplicas: 17
  observedGeneration: 1982
  readyReplicas: 17
  replicas: 17

And the following deployment:

$ kubectl get deploy scraper-go-router -o yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "16"
  creationTimestamp: "2019-11-12T02:02:17Z"
  generation: 2978
  labels:
    app: scraper-go-router
    app.kubernetes.io/name: scraper-go-router
  name: scraper-go-router
  namespace: default
  resourceVersion: "59007653"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/scraper-go-router
  uid: 74441191-04f0-11ea-955f-42010a800206
spec:
  progressDeadlineSeconds: 600
  replicas: 17
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: scraper-go-router
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: scraper-go-router
    spec:
      containers:
      - command:
        - /root/router
        image: gcr.io/basilica-211201/leadgenie-scraper-go:v0.0.3-9-g824c877
        imagePullPolicy: Always
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 17
  conditions:
  - lastTransitionTime: "2019-11-12T02:02:17Z"
    lastUpdateTime: "2019-11-14T21:37:47Z"
    message: ReplicaSet "scraper-go-router-7c564d75bd" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2019-11-18T18:46:48Z"
    lastUpdateTime: "2019-11-18T18:46:48Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 2978
  readyReplicas: 17
  replicas: 17
  updatedReplicas: 17