K
Q

Kubernetes use the same volumeMount in initContainer and Container

1/14/2022

I am trying to get a volume mounted as a non-root user in one of my containers. I'm trying an approach from this SO post using an initContainer to set the correct user, but when I try to start the configuration I get an "unbound immediate PersistentVolumneClaims" error. I suspect it's because the volume is mounted in both my initContainer and container, but I'm not sure why that would be the issue: I can see the initContainer taking the claim, but I would have thought when it exited that it would release it, letting the normal container take the claim. Any ideas or alternatives to getting the directory mounted as a non-root user? I did try using securityContext/fsGroup, but that seemed to have no effect. The /var/rdf4j directory below is the one that is being mounted as root.

Configuration:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: triplestore-data-storage-dir
  labels:
    type: local
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  storageClassName: local-storage
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /run/desktop/mnt/host/d/workdir/k8s-data/triplestore
    type: DirectoryOrCreate
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: triplestore-data-storage
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: local-storage
  volumeName: "triplestore-data-storage-dir"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: triplestore
  labels:
    app: demo
    role: triplestore
spec:
  selector:
    matchLabels:
      app: demo
      role: triplestore
  replicas: 1
  template:
    metadata:
      labels:
        app: demo
        role: triplestore
    spec:
      containers:
        - name: triplestore
          image: eclipse/rdf4j-workbench:amd64-3.5.0
          imagePullPolicy: Always
          ports:
            - name: http
              protocol: TCP
              containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 200Mi
          volumeMounts:
            - name: storage
              mountPath: /var/rdf4j
      initContainers:
        - name: take-data-dir-ownership
          image: eclipse/rdf4j-workbench:amd64-3.5.0
          command:
            - chown
            - -R
            - 100:65533
            - /var/rdf4j
          volumeMounts:
            - name: storage
              mountPath: /var/rdf4j
      volumes:
        - name: storage
          persistentVolumeClaim:
            claimName: "triplestore-data-storage"

kubectl get pvc

NAME                       STATUS   VOLUME                         CAPACITY   ACCESS MODES   STORAGECLASS    AGE
triplestore-data-storage   Bound    triplestore-data-storage-dir   10Gi       RWX            local-storage   13s

kubectl get pv

NAME                           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                              STORAGECLASS    REASON   AGE
triplestore-data-storage-dir   10Gi       RWX            Delete           Bound    default/triplestore-data-storage   local-storage            17s

kubectl get events

LAST SEEN   TYPE      REASON              OBJECT                             MESSAGE
21s         Warning   FailedScheduling    pod/triplestore-6d6876f49-2s84c    0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
19s         Normal    Scheduled           pod/triplestore-6d6876f49-2s84c    Successfully assigned default/triplestore-6d6876f49-2s84c to docker-desktop
3s          Normal    Pulled              pod/triplestore-6d6876f49-2s84c    Container image "eclipse/rdf4j-workbench:amd64-3.5.0" already present on machine
3s          Normal    Created             pod/triplestore-6d6876f49-2s84c    Created container take-data-dir-ownership
3s          Normal    Started             pod/triplestore-6d6876f49-2s84c    Started container take-data-dir-ownership
2s          Warning   BackOff             pod/triplestore-6d6876f49-2s84c    Back-off restarting failed container
46m         Normal    Pulled              pod/triplestore-6d6876f49-9n5kt    Container image "eclipse/rdf4j-workbench:amd64-3.5.0" already present on machine
79s         Warning   BackOff             pod/triplestore-6d6876f49-9n5kt    Back-off restarting failed container
21s         Normal    SuccessfulCreate    replicaset/triplestore-6d6876f49   Created pod: triplestore-6d6876f49-2s84c
21s         Normal    ScalingReplicaSet   deployment/triplestore             Scaled up replica set triplestore-6d6876f49 to 1

kubectl describe pods/triplestore-6d6876f49-tw8r8

Name:         triplestore-6d6876f49-tw8r8
Namespace:    default
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Mon, 17 Jan 2022 10:17:20 -0500
Labels:       app=demo
              pod-template-hash=6d6876f49
              role=triplestore
Annotations:  <none>
Status:       Pending
IP:           10.1.2.133
IPs:
  IP:           10.1.2.133
Controlled By:  ReplicaSet/triplestore-6d6876f49
Init Containers:
  take-data-dir-ownership:
    Container ID:  docker://89e7b1e3ae76c30180ee5083624e1bf5f30b55fd95bf1c24422fabe41ae74408
    Image:         eclipse/rdf4j-workbench:amd64-3.5.0
    Image ID:      docker-pullable://registry.com/publicrepos/docker_cache/eclipse/rdf4j-workbench@sha256:14621ad610b0d0269dedd9939ea535348cc6c147f9bd47ba2039488b456118ed
    Port:          <none>
    Host Port:     <none>
    Command:
      chown
      -R
      100:65533
      /var/rdf4j
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 17 Jan 2022 10:22:59 -0500
      Finished:     Mon, 17 Jan 2022 10:22:59 -0500
    Ready:          False
    Restart Count:  6
    Environment:    <none>
    Mounts:
      /var/rdf4j from storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s8wdv (ro)
Containers:
  triplestore:
    Container ID:
    Image:          eclipse/rdf4j-workbench:amd64-3.5.0
    Image ID:
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        100m
      memory:     200Mi
    Environment:  <none>
    Mounts:
      /var/rdf4j from storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s8wdv (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  triplestore-data-storage
    ReadOnly:   false
  kube-api-access-s8wdv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  6m24s                  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled         6m13s                  default-scheduler  Successfully assigned default/triplestore-6d6876f49-tw8r8 to docker-desktop
  Normal   Pulled            4m42s (x5 over 6m12s)  kubelet            Container image "eclipse/rdf4j-workbench:amd64-3.5.0" already present on machine
  Normal   Created           4m42s (x5 over 6m12s)  kubelet            Created container take-data-dir-ownership
  Normal   Started           4m42s (x5 over 6m12s)  kubelet            Started container take-data-dir-ownership
  Warning  BackOff           70s (x26 over 6m10s)   kubelet            Back-off restarting failed container

Solution

As it turns out the problem was that the initContainer wasn't running as root, it was running as the default user of the container, and so didn't have the permissions to run the chown command. In the linked SO comment, this was the first comment to the answer, with the response being that the initContainer ran as root - this has apparently changed in newer versions of kubernetes. There is a solution though, you can set the securityContext on the container to run as root, giving it permission to run the chown command, and that successfully allows the volume to be mounted as a non-root user. Here's the final configuration of the initContainer.

initContainers:
  - name: take-data-dir-ownership
    image: eclipse/rdf4j-workbench:amd64-3.5.0
    securityContext:
      runAsUser: 0
    command:
      - chown
      - -R
      - 100:65533
      - /var/rdf4j
    volumeMounts:
      - name: storage
        mountPath: /var/rdf4j
-- Matt McMinn
kubernetes

Similar Questions

How to identify Host cloud provider from kubernetes pods?
Connect Flask pod with mongodb pod in kubernetes.
kubernetes CI/CD: automate docker build, tag and push tasks on a CI/CD infrastructure
How to use service name instead of External IP/domain name for a service endpoint url in kubernetes
spark on k8s - Error 'Invalid initial heap size: -Xms'

1 Answer

1/21/2022

1 pod has unbound immediate PersistentVolumeClaims. - this error means the pod cannot bound to the PVC on the node where it has been scheduled to run on. This can happen when the PVC bounded to a PV that refers to a location that is not valid on the node that the pod is scheduled to run on. It will be helpful if you can post the complete output of kubectl get nodes -o wide, kubectl describe pvc triplestore-data-storage, kubectl describe pv triplestore-data-storage-dir to the question.

The mean time, PVC/PV is optional when using hostPath, can you try the following spec and see if the pod can come online:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triplestore
  labels:
    app: demo
    role: triplestore
spec:
  selector:
    matchLabels:
      app: demo
      role: triplestore
  replicas: 1
  template:
    metadata:
      labels:
        app: demo
        role: triplestore
    spec:
      containers:
      - name: triplestore
        image: eclipse/rdf4j-workbench:amd64-3.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          protocol: TCP
          containerPort: 8080
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: storage
          mountPath: /var/rdf4j
      initContainers:
      - name: take-data-dir-ownership
        image: eclipse/rdf4j-workbench:amd64-3.5.0
        imagePullPolicy: IfNotPresent
        securityContext:
          runAsUser: 0
        command:
          - chown
          - -R
          - 100:65533
          - /var/rdf4j
        volumeMounts:
          - name: storage
            mountPath: /var/rdf4j
      volumes:
        - name: storage
          hostPath:
            path: /run/desktop/mnt/host/d/workdir/k8s-data/triplestore
            type: DirectoryOrCreate
-- gohm&#39;c
Source: StackOverflow