As the state of this cluster, how exactly it was deployed as well as the output messages of kubectl, kubelet are unknown, I will try to give some troubleshooting steps and circumstances that can lead to resolving some of the problems that are encountered here.

• Kubeadm
• Flannel
• Kubelet
• Reproduction
• Additional links

Kubeadm

The first thing is provisioning process of the Kubernetes cluster with kubeadm:

Check all the requirements and steps with official documentation:

• Kubernetes.io: Install kubeadm.
• Kubernetes.io: Create cluster kubeadm

When invoking $ kubeadm init command please make sure to add parameters like:

• --apiserver-advertise-address=IP_ADDRESS_OF_MASTER_NODE
• --pod-network-cidr=POD_NETWORK_CIDR_WITH_ACCORDANCE_TO_CNI

Provisioning cluster without --pod-network-cidr parameter could lead to CNI related issues.

For Flannel default pod network cidr is 10.244.0.0/16.

After kubeadm init you need to apply one of many CNI's (like Flannel or Calico) as the kubeadm tool does not do it automatically.

Please check if all of the nodes are in Ready status with command:

$ kubectl get nodes -o wide

Output of this command should look like that:

NAME   STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
k1     Ready    master   69m   v1.17.3   10.156.0.29   <none>        Ubuntu 18.04.4 LTS   5.0.0-1031-gcp   docker://19.3.7
k2     Ready    <none>   65m   v1.17.3   10.156.0.30   <none>        Ubuntu 18.04.4 LTS   5.0.0-1031-gcp   docker://19.3.7
k3     Ready    <none>   63m   v1.17.3   10.156.0.35   <none>        Ubuntu 18.04.4 LTS   5.0.0-1031-gcp   docker://19.3.7

Additionally you can $ kubectl describe node NAME_OF_THE_NODE to get more detailed information about each of the nodes.

Flannel

There is official documentation about troubleshooting Flannel related issues: Github.com: flannel troubleshooting

The message that you received from kubelet:

Warning FailedCreatePodSandBox 34m kubelet, slave-node Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "26cdaf3170806455a4731218d20c482bb2a41ded6ef85c90b560058e332df684" network for pod "label-demo": networkPlugin cni failed to set up pod "label-demo_default" network: open /run/flannel/subnet.env: no such file or directory

Is telling that there is subnet.env file missing on a node that was supposed to schedule a sandbox pod.

Furthermore please check if Flannel pods are running correctly. You can check it with either:

• $ kubectl get pods -A
• $ kubectl get pods -n kube-system

Additionally you can check logs of this pods by running command: $ kubectl logs NAME_OF_FLANNEL_CONTAINER

Kubelet

You can check kubelet logs by (on systemd os) with:

• $ systemctl status kubelet
• $ journalctl -u kubelet

Reproduction

I've managed to reproduce error that you encountered and it was happening when:

• --pod-network-cidr=CIDR was not used with $ kubeadm init
• Flannel CNI was applied after $ kubeadm init without --pod-network-cidr

Additional links:

There is an article talking about troubleshooting networking: Kubernetes.feisky.xyz: Network.

Please let me know if you have any other questions