K
Q

Kubernetes volume emptyDir permission denied when attempting to execute file copied from init container

4/8/2021

I'm having issues with emptydir:

copying any file from an init container to an emptydir makes it unexecutable in the main container happens on 3 of my 4 nodes, no issues on single master node the master node that runs it successfully shows a difference selinux label on some files Cluster Info:

  • RHEL 7, SELinux permissive, VM's on internal datacenter
  • kubernetes 1.20, deployed by kubeadm
  • 4 node cluster (1 is a master)

Ex Pod Spec (fails on 3 nodes, successful on single master node):

apiVersion: v1
kind: Pod
metadata:
  name: emptydir-test
  namespace: default
spec:
  initContainers:
  - command:
    - "bash"
    - -c
    - "cp $(which ls) /empty-dir/empty-dir-ls ; cp $(which ls) /mem-dir/mem-ls"
    image: ubuntu
    imagePullPolicy: IfNotPresent
    name: init
    volumeMounts:
    - mountPath: /empty-dir
      name: empty-dir
    - mountPath: /mem-dir
      name: mem-dir
  containers:
  - securityContext:
      privileged: true
    command:
    - "bash"
    - -c
    - "id ; ls -alhZ /mem-dir; /mem-dir/mem-ls -alhZ /root ; ls -alhZ /empty-dir ; /empty-dir/empty-dir-ls -alhZ /root"    
    image: ubuntu
    imagePullPolicy: IfNotPresent
    name: emptydir-test
    volumeMounts:
    - mountPath: /empty-dir
      name: empty-dir
    - mountPath: /mem-dir
      name: mem-dir
  volumes:
  - emptyDir:
      medium: Memory  
    name: mem-dir
  - emptyDir: {}
    name: empty-dir

logs showing failure:

uid=0(root) gid=0(root) groups=0(root)
total 140K
drwxrwxrwt. 2 root root system_u:object_r:tmpfs_t:s0            60 Apr  8 00:32 .
drwxr-xr-x. 1 root root system_u:object_r:unlabeled_t:s0        49 Apr  8 00:32 ..
-rwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0 139K Apr  8 00:32 mem-ls
total 8.0K
drwx------. 2 root root system_u:object_r:unlabeled_t:s0   37 Apr  1 01:26 .
drwxr-xr-x. 1 root root system_u:object_r:unlabeled_t:s0   49 Apr  8 00:32 ..
-rw-r--r--. 1 root root system_u:object_r:unlabeled_t:s0 3.1K Dec  5  2019 .bashrc
-rw-r--r--. 1 root root system_u:object_r:unlabeled_t:s0  161 Dec  5  2019 .profile
total 140K
drwxrwxrwx. 2 root root system_u:object_r:container_file_t:s0   25 Apr  8 00:32 .
drwxr-xr-x. 1 root root system_u:object_r:unlabeled_t:s0        49 Apr  8 00:32 ..
-rwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0 139K Apr  8 00:32 empty-dir-ls
bash: /empty-dir/empty-dir-ls: Permission denied

logs showing success:

uid=0(root) gid=0(root) groups=0(root)
total 140K
drwxrwxrwt. 2 root root system_u:object_r:tmpfs_t:s0             60 Apr  8 00:34 .
drwxr-xr-x. 1 root root system_u:object_r:container_share_t:s0   49 Apr  8 00:34 ..
-rwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0  139K Apr  8 00:34 mem-ls
total 8.0K
drwx------. 2 root root system_u:object_r:container_share_t:s0   37 Apr  1 01:26 .
drwxr-xr-x. 1 root root system_u:object_r:container_share_t:s0   49 Apr  8 00:34 ..
-rw-r--r--. 1 root root system_u:object_r:container_share_t:s0 3.1K Dec  5  2019 .bashrc
-rw-r--r--. 1 root root system_u:object_r:container_share_t:s0  161 Dec  5  2019 .profile
total 140K
drwxrwxrwx. 2 root root system_u:object_r:container_file_t:s0    25 Apr  8 00:34 .
drwxr-xr-x. 1 root root system_u:object_r:container_share_t:s0   49 Apr  8 00:34 ..
-rwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0  139K Apr  8 00:34 empty-dir-ls
total 8.0K
drwx------. 2 root root system_u:object_r:container_share_t:s0   37 Apr  1 01:26 .
drwxr-xr-x. 1 root root system_u:object_r:container_share_t:s0   49 Apr  8 00:34 ..
-rw-r--r--. 1 root root system_u:object_r:container_share_t:s0 3.1K Dec  5  2019 .bashrc
-rw-r--r--. 1 root root system_u:object_r:container_share_t:s0  161 Dec  5  2019 .profile

I noticed the different selinux labels for the root directory between the failed (system_u:object_r:unlabeled_t:s0) and successful (system_u:object_r:container_share_t:s0) container logs. But have confirmed all nodes are in permissive mode, so not sure if selinux can/is still somehow affecting it.

Any direction or advice is greatly appreciated!

-- Eric Gagnon
kubernetes

Similar Questions

Why we have 2000 port for NodePort
how to access image content on kubernetes container init
How can I disable automatic https redirect for nginx ingress controller?
Kubernetes Init Container fail fast
How to implement graceful termination of nodes without service downtime when using cluster auto-scaler?

1 Answer

4/8/2021

After days of googling and troubleshooting, I've resolved this - the problem was: /var was mounted with noexec in fstab.

My single master node was provisioned before the others. During that time, company policy must've adopted a default noexec mount option on /var to improve security. You can either remove noexec from the /var entry in fstab (with IT's approval), or create a new mount for /var/lib/kubelet and mount that without noexec. either way, you'll need to reboot.

For anyone that runs into this issue, I also found several other potential causes:

  • ACL's - if you're using ACL's anywhere, make sure you don't have entries for /var/lib/kubelet:

    getfacl -e /var/lib/kubelet/pods

  • SELinux - if you're running in enforcing mode, check your selinux labels by -Z option with ls. These labels seem fine, not sure if there's a range of acceptable labels, but may be worth checking.

ls -alhZ /var/lib/kubelet/pods/
drwxr-xr-x. root root system_u:object_r:container_file_t:s0 .
drwxr-xr-x. root root system_u:object_r:container_file_t:s0 ..
drwxr-x---. root root system_u:object_r:container_file_t:s0 [poduid]
drwxr-x---. root root system_u:object_r:container_file_t:s0 [poduid]
# edit emptydir-test.yaml to specify node as needed, then
kubectl apply -f emptydir-test.yaml
kubectl get po -o wide # confirm it runs/fails on node
# ssh to node
ls -alhZ /var/lib/kubelet/pods/$(kubectl get po emptydir-test -o jsonpath='{.metadata.uid}')/volumes/kubernetes.io~empty-dir/empty-dir
# drwxrwxrwx. root root system_u:object_r:container_file_t:s0 .
# drwxr-xr-x. root root system_u:object_r:container_file_t:s0 ..
# -rwxr-xr-x. root root system_u:object_r:container_file_t:s0 empty-dir-ls
sudo /var/lib/kubelet/pods/$(kubectl get po emptydir-test -o jsonpath='{.metadata.uid}')/volumes/kubernetes.io~empty-dir/empty-dir/empty-dir-ls -alh /
# output should either be the result of ls -alh on / or a permission denied error
# if it's a permission denied error, you may have a noexec mount options issue
# check current mounts for noexec option:
sudo findmnt -l | grep "/var"  | grep noexec
# look for /var/lib or a mount along /var/lib/kubelet/pods with noexec option
# check fstab:
grep "/var" /etc/fstab
# /dev/mapper/vg01-lv_var  /var     xfs     nodev,noexec,nosuid   1 2
# either remove noexec from the mount options or create a new mount point for /var/lib/kubelet and omit noexec
# then reboot to take effect and run the emptydir-test pod on affected nodes

I have no doubt there's other features driving security through inaccessibility, these are just what I found while solving my issue.

-- Eric Gagnon
Source: StackOverflow