GKE Cannot pull image, even though imagesPullSecret is defined

7/10/2019

In Google Kubernetes Engine I created a POC cluster for our company which worked flawlessly. But now, when I try to create our production environment I cannot seem to get the imagesPullSecrets to work, it's the exact same credentials as in the POC, Same helm chart and the exact same regcred yaml file.

Yet i keep getting the classical:

Back-off pulling image "registry.company.co/frontend/company-web/upload": ImagePullBackOff 
  • Pulling manually on the node works with the same credentials as those that i supplied in the imagesPullSecret
  • I've tried defining the imagesPullSecret both on a chart level and on the Service Account
  • I've verified the secret format and directly copied the credentials there when trying the manual pulls
  • GKE picks up regcred and shows it in the deployment

Regcred generated by kubectl create secret docker-registry regcred --docker-server="registry.company.co" --docker-username="gitlab" --docker-password="[PASSWORD]"

regcred secret

kind: Secret
apiVersion: v1
metadata:
  name: regcred
  namespace: default
data:
  .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5jb21wYW55LmNvIjp7InVzZXJuYW1lIjoiZ2l0bGFiIiwicGFzc3dvcmQiOiJbUkVEQUNURURdIiwiYXV0aCI6IloybDBiR0ZpT2x0QmJITnZJRkpsWkdGamRHVmtYUT09In19fQ==
type: kubernetes.io/dockerconfigjson

Service Account

kind: ServiceAccount
apiVersion: v1
metadata:
  name: default
  namespace: default
secrets:
  - name: default-token-jktj5
imagePullSecrets:
  - name: regcred

Deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
    name: nfs-server
spec:
    replicas: 1
    selector:
        matchLabels:
            role: nfs-server
    template:
        metadata:
            labels:
                role: nfs-server
        spec:
            containers:
                - name: nfs-server
                  image: gcr.io/google_containers/volume-nfs:latest
                  ports:
                      - name: nfs
                        containerPort: 2049
                      - name: mountd
                        containerPort: 20048
                      - name: rpcbind
                        containerPort: 111
                  securityContext:
                      privileged: true
                  volumeMounts:
                      - mountPath: /exports
                        name: mypvc
            initContainers:
                - name: init-volume-perms
                  imagePullPolicy: Always
                  image: alpine
                  command: ["/bin/sh", "-c"]
                  args: ["mkdir /mnt/company-logos; mkdir /mnt/uploads; chown -R 1337:1337 /mnt"]
                  volumeMounts:
                      - mountPath: /mnt
                        name: mypvc
                - name: company-web-uploads
                  image: registry.company.co/frontend/company-web/uploads
                  imagePullPolicy: Always
                  volumeMounts:
                      - mountPath: /var/lib/company/web/uploads
                        subPath: uploads
                        name: mypvc
                - name: company-logos
                  image: registry.company.co/backend/pdf-service/company-logos
                  imagePullPolicy: Always
                  volumeMounts:
                      - mountPath: /var/lib/company/shared/company-logos
                        subPath: company-logos
                        name: mypvc
            volumes:
                - name: mypvc
                  gcePersistentDisk:
                      pdName: gke-nfs-disk
                      fsType: ext4

I've looked around, following different guides from the ground up to no success.

So I'm at a total loss as to what to do.

Default namespace all around

-- Martin Claesson
docker-registry
google-kubernetes-engine
kubernetes
kubernetes-helm

1 Answer

7/10/2019

It may be because of namespace issue. Can you verify a few things

  1. Are you using default namespace at both places?
  2. K8S version difference between poc and prod.
  3. Can you recreate working secret by something like kubectl get secret default-token-jktj5 -o yaml > imagepullsecret.yaml. Edit the yaml file to remove revision and other status information. Apply the same to prod
  4. I have seen this issue in GKE because of multiline secret conversion to base64. Ensure secrets are matching between environments.
-- Shubham Singh
Source: StackOverflow