Kubernetes not pulling image from private registry

5/29/2019

I am trying to deploy an image from my private registry (harbor) to my Kubernetes environment. The registry was set up successfully and already contains my image.

To give context this is my deployment file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    name: sps-app
  name: sps-app
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: sps-app
    spec:
      containers:
      - image: repo-harbor.test.com/sps_project/spsapp:23
        env:
        - name: MONGODB_URL
          value: "mongodb://mongo.default.svc.cluster.local:27017/user"
        name: sps-app
        ports:
        - containerPort: 4000
          name: sps-app
      imagePullSecrets:
      - name: harbor 

I already had create my harbor secret using the below command

kubectl create secret docker-registry harbor \
--docker-server=https://repo-harbor.test.com \
--docker-username=admin \
--docker-password='xxxxxx!'

However, when I do a kubectl apply -f of my deployment, it always goes into an image pull backoff.

Upon further investigation, I checked the logs of the pod and it states there is a x509 certification error.

Kubernetes events:

Events:
  Type     Reason          Age              From                                           Message
  ----     ------          ----             ----                                           -------
  Normal   Scheduled       3m               default-scheduler                              Successfully assigned default/private-image-test-1 to df56bd02-5e0e-4644-a565-c233ac2404fe
  Normal   Pulling         2m (x3 over 3m)  kubelet, df56bd02-5e0e-4644-a565-c233ac2404fe  pulling image "jur01-harbor.acepod.com/sps_project/spsapp:2"
  Warning  Failed          2m (x3 over 3m)  kubelet, df56bd02-5e0e-4644-a565-c233ac2404fe  Failed to pull image "jur01-harbor.acepod.com/sps_project/spsapp:2": rpc error: code = Unknown desc = Error response from daemon: Get https://jur01-harbor.acepod.com/v2/: x509: certificate signed by unknown authority
  Warning  Failed          2m (x3 over 3m)  kubelet, df56bd02-5e0e-4644-a565-c233ac2404fe  Error: ErrImagePull
  Warning  Failed          2m (x4 over 3m)  kubelet, df56bd02-5e0e-4644-a565-c233ac2404fe  Error: ImagePullBackOff
  Normal   SandboxChanged  2m (x7 over 3m)  kubelet, df56bd02-5e0e-4644-a565-c233ac2404fe  Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff         2m (x5 over 3m)  kubelet, df56bd02-5e0e-4644-a565-c233ac2404fe  Back-off pulling image "jur01-harbor.acepod.com/sps_project/spsapp:2"

At this point, I'm not sure how to resolve this. Would anyone how to resolve this?

-- adr
docker
kubernetes
ssl

1 Answer

5/30/2019

Root cause

The image registry at jur01-harbor.acepod.com uses a self-signed certificate, which Docker does not trust.

Solution

Copy the custom CA certificate presented by that image registry to all your Kubernetes nodes', in a directory called /etc/docker/certs.d/jur01-harbor.acepod.com/.

Reference: Docker docs / Test an insecure registry.

-- Antoine Cotten
Source: StackOverflow