K
Q

kube-proxy daemonset permission problems

10/3/2019

I installed a brand new 1.16.0 worker node using kubeadm and I am getting the following:

Kubernetes version: Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

OS: 18.04.3 LTS (Bionic Beaver)
Kernel:  Linux kube-node-5 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Name:           kube-proxy
Selector:       k8s-app=kube-proxy
Node-Selector:  beta.kubernetes.io/os=linux
Labels:         k8s-app=kube-proxy
Annotations:    deprecated.daemonset.template.generation: 2
Desired Number of Nodes Scheduled: 8
Current Number of Nodes Scheduled: 8
Number of Nodes Scheduled with Up-to-date Pods: 8
Number of Nodes Scheduled with Available Pods: 8
Number of Nodes Misscheduled: 0
Pods Status:  8 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           k8s-app=kube-proxy
  Service Account:  kube-proxy
  Containers:
   kube-proxy:
    Image:      k8s.gcr.io/kube-proxy:v1.15.0
    Port:       <none>
    Host Port:  <none>
    Command:
      /usr/local/bin/kube-proxy
      --config=/var/lib/kube-proxy/config.conf
      --hostname-override=$(NODE_NAME)
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/kube-proxy from kube-proxy (rw)
  Volumes:
   kube-proxy:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-proxy
    Optional:  false
   xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
   lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
Events:
  Type     Reason        Age                  From                  Message
  ----     ------        ----                 ----                  -------
  Warning  FailedCreate  3h55m                daemonset-controller  Error creating: Pod "kube-proxy-nz5bk" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h38m                daemonset-controller  Error creating: Pod "kube-proxy-l26kw" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h21m                daemonset-controller  Error creating: Pod "kube-proxy-fjcpd" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h7m                 daemonset-controller  Error creating: Pod "kube-proxy-msqnx" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h7m                 daemonset-controller  Error creating: Pod "kube-proxy-pssv5" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h7m                 daemonset-controller  Error creating: Pod "kube-proxy-59cx8" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h7m                 daemonset-controller  Error creating: Pod "kube-proxy-t9nh2" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h7m                 daemonset-controller  Error creating: Pod "kube-proxy-5hp6c" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h7m                 daemonset-controller  Error creating: Pod "kube-proxy-hbbl4" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h7m                 daemonset-controller  Error creating: Pod "kube-proxy-zph4z" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h7m                 daemonset-controller  Error creating: Pod "kube-proxy-prj9w" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  3h7m                 daemonset-controller  Error creating: Pod "kube-proxy-rhnjq" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  177m (x9 over 3h7m)  daemonset-controller  (combined from similar events): Error creating: Pod "kube-proxy-whdnm" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  166m                 daemonset-controller  Error creating: Pod "kube-proxy-2xhgt" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  149m                 daemonset-controller  Error creating: Pod "kube-proxy-zd429" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  132m                 daemonset-controller  Error creating: Pod "kube-proxy-wzn8x" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  124m                 daemonset-controller  Error creating: Pod "kube-proxy-l8csx" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  124m                 daemonset-controller  Error creating: Pod "kube-proxy-6jxpl" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  124m                 daemonset-controller  Error creating: Pod "kube-proxy-jk29x" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  124m                 daemonset-controller  Error creating: Pod "kube-proxy-p7db2" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  124m                 daemonset-controller  Error creating: Pod "kube-proxy-kf8qz" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  124m                 daemonset-controller  Error creating: Pod "kube-proxy-l5wjh" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  124m                 daemonset-controller  Error creating: Pod "kube-proxy-d8brg" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  124m                 daemonset-controller  Error creating: Pod "kube-proxy-6w2ql" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  124m                 daemonset-controller  Error creating: Pod "kube-proxy-d4n47" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
  Warning  FailedCreate  122m (x7 over 124m)  daemonset-controller  (combined from similar events): Error creating: Pod "kube-proxy-2lnpb" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy

The not so funny thing is that all the other nodes have absolutely NO problem creating the kube-proxy pods. It is only this one node that is failing with the above error.

I have tried a variety of things to fix this issue but have yet to find a solution. Previous installations using kubeadm were flawless.

I have a feeling I am missing a PodSecurityPolicy and a binding to the kube-proxy role. I am definitely missing something but I have no idea.

-- Daniel Maldonado
kube-proxy
kubeadm
kubernetes

Similar Questions

The PersistentVolume is invalid: spec: Required value: must specify a volume type
Docker container with two jar files , run on demand not as entry point
How to add insecure Docker registry certificate to kubeadm config
Making 100 concurrent HTTP Requests every two seconds, first batch takes significantly more time to complete than the rest
WSO2 ESB unable to reconnect to TIBCO JMS servers after switching from home TIBCO EMS server to secondary TIBCO EMS server

1 Answer

10/4/2019

It's very strange trying to add new node to the existing cluster from different relese. As an example for 1.1.15 The deprecated kubelet security controls AllowPrivileged please refer to release CHANGELOG-1.15.md

The deprecated kubelet security controls AllowPrivileged, HostNetworkSources, HostPIDSources, and HostIPCSources have been removed. Enforcement of these restrictions should be done through admission control (such as PodSecurityPolicy) instead

In my opinion you should remove this node (please refer before to those docs):

After that you should upgrade you cluster according to Best practices.

Please note, before you start upgrading your cluster to the v1.16.0 release: about other notable changes in the last release.

-- Hanx
Source: StackOverflow