K
Q

k8s - Failed to pull image....Error response from daemon: pull access denied for {private_repo}, repository does not exist

6/24/2020

Getting this message:

Failed to pull image....Error response from daemon: pull access denied for {private_repo}, the repository does not exist or may require 'docker login'

After deploying new helm chart using AWS ECR BUT

  1. Full private repo path is correct and image exists in ECR, in ReplicationController: private_repo/serviceXYZ:latest-develop
  2. Other pods using the SAME repo but different paths ARE working, ex: private_repo/serviceABC (their latest local images are several months old and we did deploy them recently which tells me we don't pull them locally but straight from ECR)
  3. ~/.docker/config.json shows that it's logged in
  4. There is NO secret in other services (no imagePullSecrets) which are pulled successfully

Any thoughts appreciated.

-- Anton Kim
amazon-ecr
amazon-web-services
docker
kubernetes

Similar Questions

Is there any substring function in helm chart yaml templates ?
statefulset roling update effects high availibility
Is it possible to watch all the events on a Kubernetes cluster using the python API?
Cant access kubernetes pod (minikube)
What is the recommended cert strategy for multi-master K8s clusters?

3 Answers

6/25/2020

The issue was with permissions in ECR for this particular repo, I put following permissions with "Effect": "Allow" and it worked:

"Action": [
    "ecr:BatchCheckLayerAvailability",
    "ecr:BatchGetImage",
    "ecr:CompleteLayerUpload",
    "ecr:DescribeImages",
    "ecr:GetDownloadUrlForLayer",
    "ecr:InitiateLayerUpload",
    "ecr:ListImages",
    "ecr:PutImage",
    "ecr:UploadLayerPart"
  ]

enter image description here

-- Anton Kim
Source: StackOverflow
6/24/2020

You need to authenticate to ECR to pull image. If you haven't done so, follow instructions here. Basically you get an authorization token from AWS to pass it to docker login. The account required by ECR is IAM-based and different from your local Docker account.

If you have already done that, the token may have expired. Log in again then.

The reason you don't have to do this for other pods is likely those images have been built or pulled to local so Docker doesn't have to download it (with the imagePullPolicy of the pod set to IfNotPresent as default).

-- Son Nguyen
Source: StackOverflow
6/24/2020

You didn't specify your Kubernetes resource (i.e Pod, Deployment, etc). But it's most likely as far as I can tell because either:

  • You are missing the ImagePullSecrets in the specific Pod definition:
    kubectl create secret generic regcred \
  --from-file=.dockerconfigjson=~/.docker/config.json> \
  --type=kubernetes.io/dockerconfigjson
    Then the pod:
    apiVersion: v1
kind: Pod
metadata:
  name: myservice
spec:
  containers:
  - name: yourXYZservice
    image: serviceXYZ:latest-develop
  imagePullSecrets:
  - name: regcred
  • Your private ECR repo doesn't have latest-develop tag. Before pushing, tag your image and push it with the tag:
    docker tag <image-id> serviceXYZ:latest-develop
docker push serviceXYZ:latest-develop
-- Rico
Source: StackOverflow