kubernetes fails to pull a private image [Google Cloud Container Registry, Digital Ocean]

10/2/2019

I'm trying to setup GCR with kubernetes

and getting Error: ErrImagePull Failed to pull image "eu.gcr.io/xxx/nodejs": rpc error: code = Unknown desc = Error response from daemon: pull access denied for eu.gcr.io/xxx/nodejs, repository does not exist or may require 'docker login'

Although I have setup the secret correctly in the service account, and added image pull secrets in the deployment spec

deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.18.0 (06a2e56)
  creationTimestamp: null
  labels:
    io.kompose.service: nodejs
  name: nodejs
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: nodejs
    spec:
      containers:
      - env:
        - name: MONGO_DB
          valueFrom:
            configMapKeyRef:
              key: MONGO_DB
              name: nodejs-env
        - name: MONGO_HOSTNAME
          value: db
        - name: MONGO_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongo-secret
              key: MONGO_PASSWORD
        - name: MONGO_PORT
          valueFrom:
            configMapKeyRef:
              key: MONGO_PORT
              name: nodejs-env
        - name: MONGO_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongo-secret
              key: MONGO_USERNAME
        image: "eu.gcr.io/xxx/nodejs"
        name: nodejs
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources: {}
      imagePullSecrets:
        - name: gcr-json-key
      initContainers:
        - name: init-db
          image: busybox
          command: ['sh', '-c', 'until nc -z db:27017; do echo waiting for db; sleep 2; done;']
      restartPolicy: Always
status: {}

used this to add the secret, and it said created

kubectl create secret docker-registry gcr-json-key --docker-server=eu.gcr.io  --docker-username=_json_key  --docker-password="$(cat mycreds.json)"   --docker-email=mygcpemail@gmail.com

How can I debug this, any ideas are welcome!

-- Omar S.
google-container-registry
kubernetes

2 Answers

10/7/2019

If the VM instance for pushing or pulling images and the Container Registry storage bucket are in the same Google Cloud Platform project, the Compute Engine default service account is configured with appropriate permissions to push or pull images.

If the VM instance is in a different project or if the instance uses a different service account, you must configure access to the storage bucket used by the repository.

By default, a Compute Engine VM has the read-only access scope configured for storage buckets. To push private Docker images, your instance must have read-write storage access scope configured as described in Access scopes. Please have 1 for further reference:

Please follow below table as 2:

Action Permission Role Role Title
Pull (Read Only) - storage.objects.get roles/storage.objectViewer Storage Object Viewer storage.objects.list

Also, you could share if there having any error code as you are having trouble in any steps.

-- Shafiq I
Source: StackOverflow

10/2/2019

It looks like the issue is caused by lack of permission on the related service account XXXXXXXXXXX-compute@XXXXXX.gserviceaccount.com which is missing Editor role.

Also,we need to restrict the scope to assign permissions only to push and pull images from google kubernetes engine, this account will need storage admin view permission which can be assigned by following the instructions mentioned in this article [1].

Additionally, to set the read-write storage scope when creating a Google Kubernetes Engine cluster, use the --scopes option to mention this scope "storage-rw"[2].

[1] https://cloud.google.com/container-registry/docs/access-control [2]https://cloud.google.com/container-registry/docs/using-with-google-cloud-platform#google-kubernetes-engine

-- Shafiq I
Source: StackOverflow