K
Q

Deploy Dind and secure Docker Registry on Kubernetes (colon issues)

6/30/2020

I have an issue with one of my project. Here is what I want to do :

  • Have a private docker registry on my cluster Kubernetes
  • Have a docker deamon running so that I can pull / push and build image directly inside the cluster

For this project I'm using some certificate to secure all those interactions.

1. How to reproduce :

Note: I'm working on a linux-based system

Here are the files that I'm using :

  • Deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata: 
    name: docker 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker
  template:
    metadata:
      labels:
        app: docker
    spec:
      containers: 
        - name: docker 
          image: docker:dind
          resources:
            limits:
              cpu: "0.5"
              memory: "256Mi"
            requests:
              memory: "128Mi"
          securityContext: 
              privileged: true 
          volumeMounts: 
            - name: dind-client-cert
              mountPath: /certs/client/
            - name: docker-graph-storage 
              mountPath: /var/lib/docker 
            - name: dind-registry-cert
              mountPath: >-
                /etc/docker/certs.d/registry:5000/ca.crt
          ports:
            - containerPort: 2376
      volumes: 
        - name: docker-graph-storage 
          emptyDir: {}
        - name: dind-client-cert
          persistentVolumeClaim:
              claimName: certs-client
        - name: dind-registry-cert
          secret:
            secretName: ca.crt
        - name: init-reg-vol
          secret:
            secretName: init-reg
--- 
apiVersion: apps/v1
kind: Deployment
metadata: 
    name: registry 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
    spec:
      containers: 
        - name: registry
          image: registry:2
          env:
            - name: DOCKER_TLS_CERTDIR
              value: /certs
            - name: REGISTRY_HTTP_TLS_KEY
              value: /certs/registry.pem
            - name: REGISTRY_HTTP_TLS_CERTIFICATE
              value: /certs/registry.crt
          volumeMounts: 
            - name: dind-client-cert
              mountPath: /certs/client/
            - name: dind-registry-cert
              mountPath: /certs/
            - name: registry-data
              mountPath: /var/lib/registry
          ports:
            - containerPort: 5000
      volumes:
        - name: dind-client-cert
          persistentVolumeClaim:
              claimName: certs-client
        - name: dind-registry-cert
          secret:
            secretName: registry
        - name: registry-data
          persistentVolumeClaim:
              claimName: registry-data
--- 
apiVersion: apps/v1
kind: Deployment
metadata: 
    name: client 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers: 
        - name: client 
          image: docker 
          command: ['sleep','200'] 
          resources:
            limits:
              cpu: "0.5"
              memory: "256Mi"
            requests:
              memory: "128Mi"
          env:
            - name: DOCKER_HOST
              value: tcp://docker:2376
            - name: DOCKER_TLS_VERIFY
              value: '1'
            - name: DOCKER_TLS_CERTDIR
              value: /certs
            - name: DOCKER_CERT_PATH
              value: /certs/client
            - name: REGISTRY_HTTP_TLS_CERTIFICATE
              value: /certs/registry.crt
          volumeMounts:
            - name: dind-client-cert
              mountPath: /certs/client/
              readOnly: true
            - name: dind-registry-cert
              mountPath: /usr/local/share/ca-certificate/ca.crt
              readOnly: true 
      volumes: 
        - name: dind-client-cert
          persistentVolumeClaim:
              claimName: certs-client
        - name: dind-registry-cert
          secret:
            secretName: ca.crt
  • Services.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: docker
spec:
  selector:
    app: docker
  ports:
    - name: docker
      protocol: TCP
      port: 2376
      targetPort: 2376
---
apiVersion: v1
kind: Service
metadata:
  name: registry
spec:
  selector:
    app: registry
  ports:
    - name: registry
      protocol: TCP
      port: 5000
      targetPort: 5000
  • Pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: certs-client
spec:
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Mi
status: {}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry-data
spec:
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  resources:
    limits:
      storage: 50Gi
    requests:
      storage: 2Gi
status: {}

For the cert files I have the following folder certs/ certs/client certs.d/registry:5000/ and I use these command line to generate the certs :

openssl req -newkey rsa:4096 -nodes -keyout ./certs/registry.pem -x509 -days 365 -out ./certs/registry.crt -subj "/C=''/ST=''/L=''/O=''/OU=''/CN=registry"
cp ./certs/registry.crt ./certs.d/registry\:5000/ca.crt

Then I use secrets to pass those certs inside the pods :

kubectl create secret generic registry --from-file=certs/registry.crt --from-file=certs/registry.pem
kubectl create secret generic ca.crt --from-file=certs/registry.crt

The to launch the project the following line is used :

kubectl apply -f pvc.yaml,deployment.yaml,service.yaml

2. My issues

I have a problem on my docker pods with this error :

Error: Error response from daemon: invalid volume specification: '/var/lib/kubelet/pods/727d0f2a-bef6-4217-a292-427c5d76e071/volumes/kubernetes.io~secret/dind-registry-cert:/etc/docker/certs.d/registry:5000/ca.crt:ro

So the problem seems to comme from the colon in the path name. Then I tried to escape the colon and I got this sublime error

error: error parsing deployment.yaml: error converting YAML to JSON: yaml: line 34: found unknown escape character

The real problem here is that if the folder is not named 'registry:5000' the certificat is not reconised as correct and I have a x509 error when trying to push an image from the client.

For the overall project I know that it can work like that since I already succes to deploy it localy with a docker-compose (here is the link to the github project if any of you are curious)

So I looked a bit on to it and found out that it's a recuring problem on docker (I mean on Docker Desktop for mount volumes on containers) but I can't find anything about the same issue on Kubernetes.

Do any of you have any lead / suggestion / workaround on this mater ?

As always, thanks for your times :)

------------------------------- EDIT following @HelloWorld answer -------------------------------

Thanks to the workaround with simlink the ca.cert is correctly mounted inside. Howerver since I was mounting it on the deployement that was use to run the docker deamon, the entrypoint of the container docker:dind was overwrite by the commands. For future reader here is the solution that I found : geting the entry-point.sh and running it manualy. Here is the deployement as I write those lines :

---
apiVersion: apps/v1
kind: Deployment
metadata: 
    name: docker 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker
  template:
    metadata:
      labels:
        app: docker
    spec:
      containers: 
        - name: docker 
          image: docker:dind
          resources:
            limits:
              cpu: "0.5"
              memory: "256Mi"
            requests:
              memory: "128Mi"
          securityContext: 
              privileged: true 
          command: ['sh', '-c', 'mkdir -p /etc/docker/certs.d/registry:5000 && ln -s /random/registry.crt /etc/docker/certs.d/registry:5000/ca.crt && wget https://raw.githubusercontent.com/docker-library/docker/a73d96e731e2dd5d6822c99a9af4dcbfbbedb2be/19.03/dind/dockerd-entrypoint.sh && chmod +x dockerd-entrypoint.sh && ./dockerd-entrypoint.sh']
          volumeMounts: 
            - name: dind-client-cert
              mountPath: /certs/client/
              readOnly: false 
            - name: dind-registry-cert
              mountPath: /random/
              readOnly: false 
          ports:
            - containerPort: 2376
      volumes: 
        - name: dind-client-cert
          persistentVolumeClaim:
              claimName: certs-client
        - name: dind-registry-cert
          secret:
            secretName: ca.crt

I hope it will be usefull for someone in the futur :)

-- Pacifuras
docker
docker-compose
kubernetes
registry
ssl

Similar Questions

Autoscaling daemonsets in nodepool
Getting fully_qualified_digest into yaml for deploy
Can i use k8s ingress services with external IP without set loadbalancer
How can I expose a StatefulSet service with ClusterIP None on Google Cloud Platform?
how to fix "helm go pkg: err: context deadline exceeded"

1 Answer

7/1/2020

The only thing I come up with is using symlinks. I tested it and it works. I also tried searching for better solution but didn't find anything satisfying.

Have a look at this example:

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: centos:7
    command: ['sh', '-c', 'mkdir -p /etc/docker/certs.d/registry:5000 && ln -s /some/random/path/ca.crt /etc/docker/certs.d/registry:5000/ca.crt && exec sleep 10000']
    volumeMounts:
    - mountPath: '/some/random/path'
      name: registry-cert
  volumes:
  - name: registry-cert
    secret:
      secretName: my-secret

And here is a template secret i used:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: default
type: Opaque
data:
  ca.crt: <<< some_random_Data >>>

I have mounted this secret into a /some/random/path location (without colon so it wouldn't throw errors) and created a symlink between /some/random/path/ca.crt and /etc/docker/certs.d/registry:5000/ca.crt. Of course you also need to create a dir structure before running ln -s ..., that is why I run mkdir -p ....

Let me know if you have any further questions. I'd be happy to answer them.

-- Matt
Source: StackOverflow