How to Execute kubelet from GKE

7/25/2018

In my Kubernetes cluster, when I attempt to bring up the pod, I am seeing the following error:

Pod forbidden sysctl: “***” not whitelisted

From the documentation, I need to run kubelet --experimental-allowed-unsafe-sysctls:***.

I am running in Google Kubernetes Engine, so how can I run this? I have tried using the console, but I can't see how to run this command.

-- randymay
google-kubernetes-engine

1 Answer

7/26/2018

The security.alpha.kubernetes.io/sysctls field only allows sysctls that operate within cgroup namespaces so that changing the setting does not affect unrelated pods.

As you've suggested, the default solution in Kubernetes k8s is to add the --experimental-allowed-unsafe-sysctls flag to the kubelet startup, however adding kubelet flags cannot be done in GKE because they are not accessible.

You may get around this by modifying the /etc/default/kubelet file on one of the nodes to include the experimental flag and then try to add pods to this. However this can only work as a temporary workaround as this will be overwritten by the master. Even if you edit the kubelet file, the master will restore the default configuration settings after some time.

It is not possible to override this behavior in GKE, however there are a few workarounds. You can choose either of the two solutions below. Please note that you will have to replace <ctl name> with the sysctl pod that you would like to access.

Solution:

1) The first option is to give the privilege to the container as read/write then add the command line sysctl -w [<ctl name> operation] at the bottom.

The pod yaml file will look like:

kind: Pod
metadata:
  name: sample
  labels:
    app: gpdb
spec:
  hostname: sample
  containers:
  - name: gpdb
    image: ubuntu
    securityContext:
       privileged: true
    command:
    - /bin/sh
    - -c
    - |
    sysctl -w [<ctl name> operation] && sleep 5000

2) The second option is to create a DaemonSet that runs the sysctl command manually. You can do this with a DockerFile in the following way:

Dockerfile sysctl-change

FROM alpine
CMD sysctl -w [<ctl name> operation]  ; sleep 365d

DaemonSet resource file

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: sysctl-change
spec:
  template:
    metadata:
      labels:
        name: sysctl-change
    spec:
      containers:
      - name: sysctl-change
        image: sysctl-change  # remember to set this correctly. It is the Docker image
        securityContext:
          privileged: true
      restartPolicy: Always
      nodeSelector:              # update this appropriately to the nodes 
        pool: elasticsearch

In the example above, make sure that the nodeSelector is set to an appropriate value (all the nodes you want to have the sysctl set) and that the image name is appropriate.

For more discussion on this issue, you may look at the following StackOverflow page.

-- Mahmoud Sharif
Source: StackOverflow