Environment:

Openstack with Ubuntu VM

Kuberspay install kubernetes using Cilium as network plugin.

After install:

root@admin01:/home/ubuntu# kubectl get pods -o wide --all-namespaces

NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE    IP               NODE      NOMINATED NODE   READINESS GATES
kube-system   cilium-4c4jb                            1/1     Running   0          5d4h   10.224.222.92    work003   <none>           <none>
kube-system   cilium-78tmj                            1/1     Running   0          5d4h   10.224.222.90    admin02   <none>           <none>
kube-system   cilium-8sgck                            1/1     Running   0          5d4h   10.224.222.52    admin03   <none>           <none>
kube-system   cilium-k2pck                            1/1     Running   0          5d4h   10.224.222.107   work001   <none>           <none>
kube-system   cilium-kzjxk                            1/1     Running   0          5d4h   10.224.222.64    admin01   <none>           <none>
kube-system   cilium-nfpf5                            1/1     Running   0          5d4h   10.224.222.79    work002   <none>           <none>
kube-system   cilium-operator-5d74fc4fd6-jqpjc        0/1     Pending   0          16h    <none>           <none>    <none>           <none>
kube-system   coredns-58687784f9-5j67p                0/1     Pending   0          16h    <none>           <none>    <none>           <none>
kube-system   dns-autoscaler-79599df498-pxr6x         0/1     Pending   0          16h    <none>           <none>    <none>           <none>

kubectl describe pod --namespace=kube-system cilium-operator-5d74fc4fd6-jqpjc

...
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  0/6 nodes are available: 6 node(s) had taints that the pod didn't tolerate.

Using other network plugin, I didn't find similar errors.

root@admin01:/home/ubuntu# kubectl describe node | grep -i taints

Taints:             node-role.kubernetes.io/master:NoSchedule
Taints:             node-role.kubernetes.io/master:NoSchedule
Taints:             node-role.kubernetes.io/master:NoSchedule
Taints:             node.kubernetes.io/not-ready:NoSchedule
Taints:             node.kubernetes.io/not-ready:NoSchedule
Taints:             node.kubernetes.io/not-ready:NoSchedule

Compared with the other network plugin, such as calico, /etc/cni/net.d/ just include one file: 05-cilium.conf:

root@admin01:/etc/cni/net.d# cat 05-cilium.conf
{
    "name": "cilium",
    "type": "cilium-cni"
}

extra information about node:

root@admin01:/home/ubuntu# kubectl describe node
Name:               admin01
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=admin01
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
Annotations:        io.cilium.network.ipv4-cilium-host: 10.233.64.1
                    io.cilium.network.ipv4-health-ip: 10.233.64.140
                    io.cilium.network.ipv4-pod-cidr: 10.233.64.0/24
                    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 05 Dec 2019 22:10:05 +0000
Taints:             node-role.kubernetes.io/master:NoSchedule
                    node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 05 Dec 2019 22:12:23 +0000   Thu, 05 Dec 2019 22:12:23 +0000   CiliumIsUp                   Cilium is running on this node
  MemoryPressure       False   Thu, 12 Dec 2019 02:12:56 +0000   Thu, 05 Dec 2019 22:09:59 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 12 Dec 2019 02:12:56 +0000   Thu, 05 Dec 2019 22:09:59 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 12 Dec 2019 02:12:56 +0000   Thu, 05 Dec 2019 22:09:59 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                False   Thu, 12 Dec 2019 02:12:56 +0000   Thu, 05 Dec 2019 22:09:59 +0000   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  10.224.222.64
  Hostname:    admin01
Capacity:
 cpu:                2
 ephemeral-storage:  81120644Ki
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             4039464Ki
 pods:               110
Allocatable:
 cpu:                1800m
 ephemeral-storage:  74760785387
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             3437064Ki
 pods:               110
System Info:
 Machine ID:                 051d190f0ef34b08ac10f08827a564ed
 System UUID:                051D190F-0EF3-4B08-AC10-F08827A564ED
 Boot ID:                    48689d4e-bba2-4f69-85fd-4ec4b12981c5
 Kernel Version:             4.15.0-42-generic
 OS Image:                   Ubuntu 18.04.1 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://18.9.7
 Kubelet Version:            v1.16.3
 Kube-Proxy Version:         v1.16.3
PodCIDR:                     10.233.64.0/24
PodCIDRs:                    10.233.64.0/24
Non-terminated Pods:         (6 in total)
  Namespace                  Name                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                               ------------  ----------  ---------------  -------------  ---
  kube-system                cilium-kzjxk                       100m (5%)     500m (27%)  64M (1%)         500M (14%)     6d4h
  kube-system                kube-apiserver-admin01             250m (13%)    0 (0%)      0 (0%)           0 (0%)         6d4h
  kube-system                kube-controller-manager-admin01    200m (11%)    0 (0%)      0 (0%)           0 (0%)         6d4h
  kube-system                kube-proxy-r9mk6                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         44h
  kube-system                kube-scheduler-admin01             100m (5%)     0 (0%)      0 (0%)           0 (0%)         6d4h
  kube-system                nodelocaldns-6wj8x                 100m (5%)     0 (0%)      70Mi (2%)        170Mi (5%)     6d4h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests        Limits
  --------           --------        ------
  cpu                750m (41%)      500m (27%)
  memory             137400320 (3%)  678257920 (19%)
  ephemeral-storage  0 (0%)          0 (0%)
Events:              <none>

Whether Cilium need more CPU resource than other network plugin? Whether should I assign more CPU resource to kubernetes administrator nodes?