K
Q

Make systemctl work from inside a container in a debian stretch image

1/7/2019

Purpose - What do I want to achieve?

I want to access systemctl from inside a container running a kubernetes node (ami: running debian stretch)

Working setup:

  • Node AMI: kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17

  • Node Directories Mounted in the container to make systemctl work:

    • /var/run/dbus
    • /run/systemd
    • /bin/systemctl
    • /etc/systemd/system

Not Working setup:

  • Node AMI: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17

  • Node Directories Mounted in the container to make systemctl work:

    • /var/run/dbus
    • /run/systemd
    • /bin/systemctl
    • /etc/systemd/system

Debugging in an attempt to solve the problem

To debug this issue with the debian-stretch image not supporting systemctl with the same mounts as debian-jessie

1) I began by spinning up a nginx deployment by mounting the above mentioned volumes in it

kubectl apply -f https://k8s.io/examples/application/deployment.yaml

kubectl exec -it nginx-deployment /bin/bash

root@nginx-deployment-788f65877d-pzzrn:/# systemctl
systemctl: error while loading shared libraries: libsystemd-shared- 
232.so: cannot open shared object file: No such file or directory

2) As the above issue showed the file libsystemd-shared-232.so not found. I found the actual path by looking into the node.

admin@ip-10-0-20-11:~$ sudo find / -iname 'libsystemd-shared-232.so'
/lib/systemd/libsystemd-shared-232.so

3) Mounted the /lib/systemd in the nginx pod and ran the systemctl again

 kubectl exec -it nginx-deployment /bin/bash

 root@nginx-deployment-587d866f54-ghfll:/# systemctl
 systemctl: error while loading shared libraries: libcap.so.2:cannot 
 open shared object file: No such file or directory

4) Now the systemctl was failing with a new so missing error

root@nginx-deployment-587d866f54-ghfll:/# systemctl
systemctl: error while loading shared libraries: libcap.so.2: cannot 
open shared object file: No such file or directory

5) To solve the above error i again searched the node for libcap.so.2 Found it in the below path.

admin@ip-10-0-20-11:~$ sudo find / -iname 'libcap.so.2'
/lib/x86_64-linux-gnu/libcap.so.2

6) Seeing the above directory not mounted in my pod. I mounted the below path in the nginx pod.

/lib/x86_64-linux-gnu mounted in the nginx pod(deployment)

7) The nginx pod is not able to come up after adding the above mount. Getting the below error:

$ k logs nginx-deployment-f9c5ff956-b9wn5
standard_init_linux.go:178: exec user process caused "no such file 
or directory"

Please suggest how to debug further. And what all mounts are required to make systemctl work from inside a container in a debian stretch environment.

Any pointers to take the debugging further could be helpful.

-- Alok Kumar Singh
debian
docker
kops
kubernetes
systemd

Similar Questions

Master Public name, Internal name, etcd member pre-built DNS records doesn't resolve to Master external IP address on GCE
Can not connect to service in 192.168.99.101
Knative pod http request
Do we need to have all ports opened between kubernetes nodes
Entity Framework Core leaving many connections in sleeping status
Custom nginx.conf from ConfigMap in Kubernetes
K8s pods going in Evicted state while deploying Spring boot application

3 Answers

3/4/2020

I had a similar problem where one of the lines in my Dockerfile was: RUN apt-get install -y --reinstall systemd but after docker restart, when I tried to use systemctl command. The output was: Failed to connect to bus: No such file or directory. I solved this issue by adding following to my docker-compose.yml: volumes: - "/sys/fs/cgroup:/sys/fs/cgroup:ro" It can be done also by: sudo docker run -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro {other options}

-- Anuraag Rijal
Source: StackOverflow
1/7/2019

Rather than mounting some of the library files from the host you can just install systemd in the container.

$ apt-get -y install systemd

Now, that won't necessarily make systemctl run. You will need systemd to be running in your container which is spawned by /sbin/init on your system. /sbin/init needs to run as root so essentially you would have to run this with the privileged flag in the pod or container security context on Kubernetes. Now, this is insecure and there is a long history about running systemd in a container where the Docker folks were mostly against it (security) and the Red Hat folks said that it was needed.

Nevertheless, the Red Hat folks figured out a way to make it work without the unprivileged flag. You need:

  • /run mounted as a tmpfs in the container.
  • /sys/fs/cgroup mounted as read-only is ok.
  • /sys/fs/cgroup/systemd/ mounted as read/write.
  • Use for STOPSIGNAL SIGRTMIN+3

In Kubernetes you need an emptyDir to mount a tmpfs. The others can be mounted as host volumes.

-- Rico
Source: StackOverflow
1/7/2019

After mounting your host's /lib directory into the container, your Pod most probably will not start because the Docker image's /lib directory contained some library needed by the Nginx server that should start in that container. By mounting /lib from the host, the libraries required by Nginx will not be accessible any more. This will result in a No such file or directory error when trying to start Nginx.

To make systemctl available from within the container, I would suggest simply installing it within the container, instead of trying to mount the required binaries and libraries into the container. This can be done in your container's Dockerfile:

FROM whatever

RUN apt-get update && apt-get install systemd

No need to mount /bin/systemd or /lib/ with this solution.

-- helmbert
Source: StackOverflow