k3s pods are not mounting secrets defined in helm deployment's imagePullSecrets

11/24/2020

I am creating a deployment in circlci that deploys my containerized application to a k3s server I have set up. I have set up a secret using the commands found here.

The secret is created using the command:

kubectl create secret docker-registry regkeyname --docker-server=https://index.docker.io/v1/ \
 --docker-username=username  \
 --docker-password=password  \
 --docker-email=my@email.com \
 --namespace=external

My secret is as follows when running kubectl get secret regkeyname --namespace=external --output=yaml:

apiVersion: v1
data:
  .dockerconfigjson: secretbase64stuff
kind: Secret
metadata:
  creationTimestamp: "2020-11-24T13:11:07Z"
  managedFields:
    - apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          .: {}
          f:.dockerconfigjson: {}
        f:type: {}
      manager: kubectl
      operation: Update
      time: "2020-11-24T13:11:07Z"
  name: regkeyname
  namespace: external
  resourceVersion: "16929381"
  selfLink: /api/v1/namespaces/external/secrets/regkeyname
  uid: 51b87508-9cf2-490b-b871-0b5a342ab64c
type: kubernetes.io/dockerconfigjson

I'm using helm to deploy my application and the Deployment looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.labels.app }}
  labels:
    app: {{ .Values.labels.app }}
spec:
  selector:
    matchLabels:
      app: {{ .Values.labels.app }}
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: {{ .Values.labels.app }}
        env: {{ .Values.labels.env }}
    spec:
      imagePullSecrets:
        - name: regkeyname
      containers:
        - name: my-service
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          imagePullPolicy: {{ .Values.image.imagePullPolicy }}
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 5
            successThreshold: 1

after deploying however, the images fail to pull and it appears that my secret "regkeyname" is not used/mounted in the pods. the result is as follows:

Name:           my-service-856454c6cd-qcp7w
Namespace:      external
Priority:       0
Node:           worker-2/192.168.1.13
Start Time:     Tue, 24 Nov 2020 07:20:08 -0600
Labels: app=my-service
  env=development
  pod-template-hash=856454c6cd
Annotations:    <none>
Status:         Pending
IP:             10.42.2.196
Controlled By:  ReplicaSet/my-service-856454c6cd
Containers:
  auth-service:
    Container ID:
    Image:          my-repo/my-service:latest
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Readiness:      http-get http://:8080/health delay=10s timeout=1s period=10s #success=1 #failure=5
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-l9b4k (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-l9b4k:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-l9b4k
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
  node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                  Message
  ----     ------     ----                   ----                  -------
  Normal   Scheduled  <unknown>              default-scheduler     Successfully assigned external/auth-service-856454c6cd-qcp7w to worker-2
  Normal   Pulling    32m (x4 over 34m)      kubelet, worker-2  Pulling image "my-repo/my-service:latest"
  Warning  Failed     32m (x4 over 34m)      kubelet, worker-2  Failed to pull image "my-repo/my-service:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/my-repo/my-service:latest": failed to resolve reference "docker.io/my-repo/my-service:latest": failed to do request: Head https://registry-1.docker.io/v2/my-repo/my-service/manifests/latest: dial tcp: lookup registry-1.docker.io: Try again
  Warning  Failed     32m (x4 over 34m)      kubelet, worker-2  Error: ErrImagePull
  Warning  Failed     31m (x6 over 34m)      kubelet, worker-2  Error: ImagePullBackOff
  Normal   BackOff    3m54s (x127 over 34m)  kubelet, worker-2  Back-off pulling image "my-repo/my-service:latest"

I had this working when running locally with kubernetes so I am assuming the issue must have something to do either with k3s or the fact that now the server is remote rather than local. Any insight would be greatly appreciated. Thanks in advance!

-- Marcus Ruddick
docker
k3s
kubernetes
kubernetes-helm
kubernetes-secrets

1 Answer

11/24/2020

The controller is trying to pull image from the official docker registry:

failed to resolve reference "docker.io/my-repo/my-service:latest"

While creating the imagePullSecret, make sure that you put the correct URL (ie. the URL for your private registry) for performing authentication and pulling image.

$ cat ~/.docker/config.json 
{
	"auths": {
		"https://index.docker.io/v1/": { # <------ change here
			"auth": "..........="
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.5 (linux)"
	}
-- Kamol Hasan
Source: StackOverflow