How to make kubernetes docker secret work?

10/26/2018

Data showing "xxx" has been masked.

Problem description:

Success Scenario: When i make my image public in docker registry, my pod is getting created successfully.

Failure Scenario: When i make my image private in docker registry. My image pull fails on kubernetes cluster.

Please details below and help.

I have my image published to docker registry.

Following is my kubernetes secret:

c:\xxxxxxx\temp>kubectl get secret regcredx -o yaml
apiVersion: v1
data:
  .dockerconfigjson: xxxxxx
kind: Secret
metadata:
  creationTimestamp: 2018-10-25T21:38:18Z
  name: regcredx
  namespace: default
  resourceVersion: "1174545"
  selfLink: /api/v1/namespaces/default/secrets/regcredx
  uid: 49a71ba5-d89e-11e8-8bd2-005056b7126c
type: kubernetes.io/dockerconfigjson

Here is my pod.yaml file:

apiVersion: v1
kind: Pod
metadata:
  name: whatever
spec:
  containers:
    - name: whatever
      image: xxxxxxxxx/xxxxxx:123
      imagePullPolicy: Always
      command: [ "sh", "-c", "tail -f /dev/null" ]
  imagePullSecrets:
    - name: regcredx

Here is my pod config in cluster:

c:\Sharief\temp>kubectl get pod whatever -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 100.96.1.81/32
  creationTimestamp: 2018-10-26T20:49:11Z
  name: whatever
  namespace: default
  resourceVersion: "1302024"
  selfLink: /api/v1/namespaces/default/pods/whatever
  uid: 9783b81f-d960-11e8-94ca-005056b7126c
spec:
  containers:
  - command:
    - sh
    - -c
    - tail -f /dev/null
    image: xxxxxxxxx/xxxxxxx:123
    imagePullPolicy: Always
    name: whatever
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-4db4c
      readOnly: true
   dnsPolicy: ClusterFirst
  imagePullSecrets:
  - name: regcredx
  nodeName: xxxx-pvt
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-4db4c
    secret:
      defaultMode: 420
      secretName: default-token-4db4c
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2018-10-26T20:49:33Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2018-10-26T20:49:33Z
    message: 'containers with unready status: [whatever]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2018-10-26T20:49:11Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: xxxxxxxxx/xxxxxxx:123
    imageID: ""
    lastState: {}
    name: whatever
    ready: false
    restartCount: 0
    state:
      waiting:
        message: Back-off pulling image "xxxxxxxxx/xxxxxxx:123"
        reason: ImagePullBackOff
  hostIP: xx.xxx.xx.xx
  phase: Pending
  podIP: xx.xx.xx.xx
  qosClass: BestEffort
  startTime: 2018-10-26T20:49:33Z

Here is my pod discription:

c:\xxxxxxx\temp>kubectl describe pod whatever
Name:               whatever
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               co2-vmkubwrk01company-pvt/xx.xx.xx.xx
Start Time:         Fri, 26 Oct 2018 15:49:33 -0500
Labels:             <none>
Annotations:        cni.projectcalico.org/podIP=xxx.xx.xx.xx/xx
Status:             Pending
IP:                 xxx.xx.x.xx
Containers:
  whatever:
    Container ID:
    Image:         xxxxxxxxx/xxxxxxx:123
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      tail -f /dev/null
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4db4c (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  default-token-4db4c:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-4db4c
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                 From                                Message
  ----     ------                 ----                ----                                -------
  Normal   Scheduled              27m                 default-scheduler                   Successfully assigned whatever to xxx
  Normal   SuccessfulMountVolume  26m                 kubelet, co2-vmkubwrk01company-pvt  MountVolume.SetUp succeeded for volume "default-token-4db4c"
  Normal   Pulling                25m (x4 over 26m)   kubelet, co2-vmkubwrk01company-pvt  pulling image "xxxxxxxxx/xxxxxxx:123"
  Warning  Failed                 25m (x4 over 26m)   kubelet, co2-vmkubwrk01company-pvt  Failed to pull image "xxxxxxxxx/xxxxxxx:123": rpc error: code = Unknown desc = repository docker.io/xxxxxxxxx/xxxxxxx not found: does not exist or no pull access
  Warning  Failed                 25m (x4 over 26m)   kubelet, co2-vmkubwrk01company-pvt  Error: ErrImagePull
  Normal   BackOff                16m (x41 over 26m)  kubelet, co2-vmkubwrk01company-pvt  Back-off pulling image "xxxxxxxxx/xxxxxxx:123"
  Warning  Failed                 1m (x106 over 26m)  kubelet, co2-vmkubwrk01company-pvt  Error: ImagePullBackOff
-- shaiksphere
docker
kubernetes
kubernetes-helm

1 Answer

10/26/2018

Kubernetes could not find your repository , the image path is wrong , you need to fix this:

image: xxxxxxxxx/xxxxxx:123

One thing you can try to test the assumption that pre-pull the image on the node on which the deployment is going to happen. do docker images , note the correct uri/repo:tag and update it in you deployment.

-- Ijaz Ahmad Khan
Source: StackOverflow