Operation not permitted when performing a traceroute from a container deployed in Kubernetes [Linux capabilities]

4/5/2020

Trying to understand security context and capabilities in Kubernetes, I created following pod description:

echo 'apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  securityContext:
    runAsUser: 1000
  containers:
  - name: busy
    image: busybox
    command:
     - sleep
     - "3600"
    securityContext:
      runAsUser: 2000
      capabilities : 
        add: ["NET_ADMIN", "SYS_TIME"]
' > app.yaml

Where I am adding NET_ADMIN and SYS_TIME capabilities.

Given Linux capabilities man page: http://man7.org/linux/man-pages/man7/capabilities.7.html

I would expect to be able to perform a traceroute or set the date:

$ kubectl exec -it app -- traceroute google.fr
traceroute: socket: Operation not permitted
command terminated with exit code 1

$ kubectl exec -it app -- /bin/sh
/ $ date --set="10:00:00"
date: can't set date: Operation not permitted

Since correct capabilities are set, I found strange that those operations are not permitted. Is this actually expected?

-- scoulomb
docker
kubernetes

1 Answer

4/23/2020

In your example you are using Busybox.

Coming in somewhere between 1 and 5 Mb in on-disk size (depending on the variant), BusyBox is a very good ingredient to craft space-efficient distributions. BusyBox combines tiny versions of many common UNIX utilities into a single small executable. It provides replacements for most of the utilities you usually find in GNU fileutils, shellutils, etc. The utilities in BusyBox generally have fewer options than their full-featured GNU cousins; however, the options that are included provide the expected functionality and behave very much like their GNU counterparts. BusyBox provides a fairly complete environment for any small or embedded system.

I have tried to achieve what you want in many verious scenarios. Honestly, examples you choose to test securityContext here are not the best. I will post quite detailed information why.

To run traceroute or set date on busybox you need proper privileges. If you would use default busybox pod with root priviligies like below example It will works as expected.

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: busy
    image: busybox
    command:
     - sleep
     - "3600"

$ kubectl exec -ti app -- traceroute bbc.com
traceroute to bbc.com (151.101.128.81), 30 hops max, 46 byte packets
 1  10.32.1.1 (10.32.1.1)  0.006 ms  0.007 ms  0.003 ms
 2  216.239.48.36 (216.239.48.36)  5.476 ms  216.239.48.74 (216.239.48.74)  5.361 ms  216.239.48.36 (216.239.48.36)  4.669 ms
 ...
$ kubectl exec -ti app -- ping bbc.com
PING bbc.com (151.101.0.81): 56 data bytes
64 bytes from 151.101.0.81: seq=0 ttl=54 time=6.246 ms
64 bytes from 151.101.0.81: seq=1 ttl=54 time=6.081 ms

To run traceroute you need sudoprivileges`. For detailed information please check docs about traceroute on busybox.

As was mentioned in Kubernetes documentation regarding securityContext, in your YAML configuration you have set:

echo 'apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  securityContext:
    runAsUser: 1000 ## All containers in this pod will be run as user 1000
  containers:
  - name: busy
    image: busybox
    command:
     - sleep
     - "3600"
    securityContext:
      runAsUser: 2000 ## as you specified here, as default you will enter to this container as user with ID 2000
      capabilities : 
        add: ["NET_ADMIN", "SYS_TIME"]

In example above you have set runAsUser: 1000 which means that each container in this pod, default login will be as user 1000. Under container spec, you have set runAsUser: 2000 which means this specific container, as default will be logged in as user 2000.

To explain who is user 1000 please check this docs. In short this number is

Notice how the root user has the UID of 0. Most Linux distributions reserve the first 100 UIDs for system use. New users are assigned UIDs starting from 500 or 1000. For example, new users in Ubuntu start from 1000

Next thing I want to mention is Linux capabilities:

Output from BusyBox:

$ kubectl exec -ti app /bin/sh
/ # capsh --print
/bin/sh: capsh: not found

Output from Ubuntu:

$ kubectl exec -ti ubuntu /bin/bash
root@ubuntu:/# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
...
root@ubuntu:/# capsh --print
Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap

If you want to use capabilites, you will not achieve it using Busybox. If you want some extra information about traceroute for linxu, please check this link.

As last test for using traceroute I've created random user in container.

Ubuntu (default ubuntu image didn't have traceroute, needed to install it. apt-get update to update repository and then apt-get install traceroute:

$ kubectl exec -ti ubuntu /bin/sh
# whoami
root
# traceroute bbc.com
traceroute to bbc.com (151.101.0.81), 30 hops max, 60 byte packets
 1  10.32.1.1 (10.32.1.1)  0.032 ms  0.008 ms  0.007 ms
 2  209.85.253.197 (209.85.253.197)  6.294 ms 216.239.48.74 (216.239.48.74)  5.613 ms 216.239.48.36 (216.239.48.36)  5.335 ms
# useradd -m test    
# passwd test
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
# su test
$ whoami
test
$ id  
uid=1000(test) gid=1000(test) groups=1000(test)
$ traceroute bbc.com
traceroute to bbc.com (151.101.64.81), 30 hops max, 60 byte packets
 1  10.32.1.1 (10.32.1.1)  0.034 ms  0.008 ms  0.008 ms
 2  216.239.48.36 (216.239.48.36)  5.515 ms 216.239.51.111 (216.239.51.111)  5.494 ms 216.239.48.36 (216.239.48.36)  5.591 ms

Busybox:

$ kubectl exec -ti app /bin/sh
/ # whoami
root
/ # traceroute bbc.com
traceroute to bbc.com (151.101.192.81), 30 hops max, 46 byte packets
 1  10.32.1.1 (10.32.1.1)  0.005 ms  0.006 ms  0.003 ms
 2  216.239.48.36 (216.239.48.36)  5.453 ms  216.239.48.74 (216.239.48.74)  4.812 ms  209.85.252.4 (209.85.252.4)  6.787 ms
/ # adduser test
Changing password for test
New password:
Retype password:
passwd: password for test changed by root
/ # su test
/ $ whoami
test
/ $ id
uid=1000(test) gid=1000(test) groups=1000(test)
/ $ traceroute bbc.com
traceroute: socket: Operation not permitted

In short story, to execute traceroute in Busybox you need have root privileges. To run traceroute on ubuntu you have to pre-install traceroute command.

Regarding changing date in container, please check this tread.

-- PjoterS
Source: StackOverflow