K
Q

Question

Error while creating High Availability Kubernetes cluster with Kubeadm

6/24/2019

I am creating Kubernetes Cluster inside VM (image that I used inside VM is CentOS 7 CIS Hardened) with kubeadm.

I am following official documentation of creating High Availability Cluster with Kubeadm https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/#first-steps-for-both-methods

Steps that I have done so far:

Installed Docker and Enabled Docker

yum update
yum install docker
sudo systemctl start docker
sudo systemctl status docker
systemctl enable docker.service

Change these values to 1 instead of 0

nano /proc/sys/net/bridge/bridge-nf-call-iptables
nano /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

Check the SELinux Status

sestatus

Disable SELinux

setenforce 0

To permanently disable SELinux on your CentOS 7 system, follow the steps below:

nano /etc/selinux/config

set the SELINUX mod to disabled:

Disable Swap: swapoff -a sed -e '/swap/ s/^#*/#/' -i /etc/fstab mount -a

Install Kubelet, Kubeadm, Kubectl at once

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF

Installing kubelet, kubeadm, kubectl in one command

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet && systemctl start kubelet
systemctl daemon-reload

After doing this i created a file called kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "LoadBalancerIP"

kubeadm init --config=kubeadm-config.yaml --upload-certs

[init] Using Kubernetes version: v1.15.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [vm3 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.0.6 52.174                                                                           .21.65]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [vm3 localhost] and IPs [10.0.0.6 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [vm3 localhost] and IPs [10.0.0.6 127.0.0.1 ::1]
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'

- 'journalctl -xeu kubelet'

But,

systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Mon 2019-06-24 07:23:57 UTC; 1h 44min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 29985 (kubelet)
   CGroup: /system.slice/kubelet.service
           └─29985 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cg...

Jun 24 09:08:21 vm3 kubelet[29985]: E0624 09:08:21.671662   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:08:21 vm3 kubelet[29985]: E0624 09:08:21.771927   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:08:21 vm3 kubelet[29985]: E0624 09:08:21.872163   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:08:21 vm3 kubelet[29985]: E0624 09:08:21.972355   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:08:22 vm3 kubelet[29985]: E0624 09:08:22.072534   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:08:22 vm3 kubelet[29985]: E0624 09:08:22.172783   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:08:22 vm3 kubelet[29985]: E0624 09:08:22.272966   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:08:22 vm3 kubelet[29985]: E0624 09:08:22.373141   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:08:22 vm3 kubelet[29985]: E0624 09:08:22.473314   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:08:22 vm3 kubelet[29985]: E0624 09:08:22.573542   29985 kubelet.go:2248] node "vm3" not found
[root@vm3 vm2]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-06-24 07:12:39 UTC; 1h 56min ago
     Docs: http://docs.docker.com
 Main PID: 28682 (dockerd-current)
   CGroup: /system.slice/docker.service
           ├─28682 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --u...
           ├─28688 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/dock...
           ├─30073 /usr/bin/docker-containerd-shim-current 6873dad40999d42255b3ffb78f1212409b8ae01450853e33b928f67a3b5f9116 /var/run/docker/libcontainerd/6873dad40999d42255b3ffb78f12...
           ├─30109 /usr/bin/docker-containerd-shim-current eb3945f99d1a5b7aa1c07086a2fed12a60afa7ffec3a89127fe51615f11a9621 /var/run/docker/libcontainerd/eb3945f99d1a5b7aa1c07086a2fe...
           ├─30170 /usr/bin/docker-containerd-shim-current 930354300ef22146adf972f098562336ec2dcf274808cc18d8db6d22ec425673 /var/run/docker/libcontainerd/930354300ef22146adf972f09856...
           ├─30237 /usr/bin/docker-containerd-shim-current dfdb99a011f6762d86986c211663b162c43dd1c20f99f63fd855d87242f2f617 /var/run/docker/libcontainerd/dfdb99a011f6762d86986c211663...
           └─30366 /usr/bin/docker-containerd-shim-current 7f13a2b265d9733d8db34aa7acfe7f320c3ef2b5d523146b2ffeefa6b3bf666c /var/run/docker/libcontainerd/7f13a2b265d9733d8db34aa7acfe...

Jun 24 09:09:26 vm3 dockerd-current[28682]: E0624 09:09:26.396927       1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: Get htt... i/o timeout
Jun 24 09:09:26 vm3 dockerd-current[28682]: I0624 09:09:26.398140       1 trace.go:81] Trace[922903595]: "Reflector k8s.io/client-go/informers/factory.go:133 ListAndWatch...000377891s):
Jun 24 09:09:26 vm3 dockerd-current[28682]: Trace[922903595]: [30.000377891s] [30.000377891s] END
Jun 24 09:09:26 vm3 dockerd-current[28682]: E0624 09:09:26.398164       1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolume... i/o timeout
Jun 24 09:09:26 vm3 dockerd-current[28682]: I0624 09:09:26.399536       1 trace.go:81] Trace[1309975098]: "Reflector k8s.io/client-go/informers/factory.go:133 ListAndWatc...000249191s):
Jun 24 09:09:26 vm3 dockerd-current[28682]: Trace[1309975098]: [30.000249191s] [30.000249191s] END
Jun 24 09:09:26 vm3 dockerd-current[28682]: E0624 09:09:26.399556       1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicationContr... i/o timeout
Jun 24 09:09:26 vm3 dockerd-current[28682]: I0624 09:09:26.400806       1 trace.go:81] Trace[1990792995]: "Reflector k8s.io/client-go/informers/factory.go:133 ListAndWatc...000302791s):
Jun 24 09:09:26 vm3 dockerd-current[28682]: Trace[1990792995]: [30.000302791s] [30.000302791s] END
Jun 24 09:09:26 vm3 dockerd-current[28682]: E0624 09:09:26.400824       1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: Ge... i/o timeout
Hint: Some lines were ellipsized, use -l to show in full.


----


journalctl -xeu kubelet
Jun 24 09:10:47 vm3 kubelet[29985]: E0624 09:10:47.173524   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:47 vm3 kubelet[29985]: E0624 09:10:47.273712   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:47 vm3 kubelet[29985]: E0624 09:10:47.373946   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:47 vm3 kubelet[29985]: E0624 09:10:47.474120   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:47 vm3 kubelet[29985]: E0624 09:10:47.574342   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:47 vm3 kubelet[29985]: E0624 09:10:47.674566   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:47 vm3 kubelet[29985]: E0624 09:10:47.774791   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:47 vm3 kubelet[29985]: E0624 09:10:47.874989   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:47 vm3 kubelet[29985]: E0624 09:10:47.975152   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: I0624 09:10:48.032756   29985 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.037735   29985 pod_workers.go:190] Error syncing pod aadedc1082b1437c80407350a8e8e7da ("etcd-vm3_kube-system(aadedc1082b1437c80407350a8e8e7da)"), skipping: failed to "StartContainer" for "etcd" with CrashLoopB
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.075398   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.175561   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.275773   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.375943   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.476103   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.576329   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: I0624 09:10:48.640424   29985 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Jun 24 09:10:48 vm3 kubelet[29985]: I0624 09:10:48.644654   29985 kubelet_node_status.go:72] Attempting to register node vm3
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.676515   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.776713   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.876884   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:48 vm3 kubelet[29985]: E0624 09:10:48.977079   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.077281   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.177529   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.277724   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: W0624 09:10:49.278257   29985 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.341494   29985 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.377962   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.478116   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.578271   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.678475   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.778660   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.878921   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:49 vm3 kubelet[29985]: E0624 09:10:49.979137   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.079316   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.179554   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.279752   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.379896   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.480148   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.580264   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.680496   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.780740   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.804619   29985 kubelet.go:2169] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.880896   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:50 vm3 kubelet[29985]: E0624 09:10:50.981216   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.081409   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.181648   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.281779   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.381904   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.482159   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.582329   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.682502   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.782682   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.882868   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:51 vm3 kubelet[29985]: E0624 09:10:51.983112   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:52 vm3 kubelet[29985]: E0624 09:10:52.083299   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:52 vm3 kubelet[29985]: E0624 09:10:52.183510   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:52 vm3 kubelet[29985]: E0624 09:10:52.283712   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:52 vm3 kubelet[29985]: E0624 09:10:52.383869   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:52 vm3 kubelet[29985]: E0624 09:10:52.484123   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:52 vm3 kubelet[29985]: E0624 09:10:52.584315   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:52 vm3 kubelet[29985]: E0624 09:10:52.684482   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:52 vm3 kubelet[29985]: E0624 09:10:52.784715   29985 kubelet.go:2248] node "vm3" not found
Jun 24 09:10:52 vm3 kubelet[29985]: E0624 09:10:52.884935   29985 kubelet.go:2248] node "vm3" not found

I needed to create HA Kubnetes cluster inside CentOS image. But I stuck at this point.

-- Tahir Javed

azure

devops

docker

high-availability

kubernetes

1 Answer

11/3/2019

You should use --node-name with kubeadm init:

$ sudo kubeadm init --node-name master1

The flag passes the appropriate --hostname-override value to the kubelet: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#setting-the-node-name

As you are using a config file, add a field name in nodeRegistration paragraph in InitConfiguration:

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  name: "master1"

You can refer to the following documentation: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#hdr-Kubeadm_init_configuration_types

-- Stéphane Beuret

Source: StackOverflow

KQ