Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container

12/23/2020

We are trying to create POD but the Pod's status struck at ContainerCreating for long time.

This is the output we got after running the command: kubectl describe pod

Name:           demo-6c59fb8f77-9x6sr
Namespace:      default
Priority:       0
Node:           k8-slave2/10.0.0.5
Start Time:     Wed, 23 Dec 2020 10:16:23 +0000
Labels:         app=demo
                pod-template-hash=6c59fb8f77
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/demo-6c59fb8f77
Containers:
  private-docker-registry:
    Container ID:
    Image:          private-docker-registry:5000/mahin/mof-docker-demo:v1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p94zw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-p94zw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p94zw
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  

  Events:
      Type     Reason                  Age                  From               Message
      ----     ------                  ----                 ----               -------
      Normal   Scheduled               10m                  default-scheduler  Successfully assigned default/demo-6c59fb8f77-9x6sr to k8-slave2
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "95e72bfc6f6c13de7f5c96eb76b012c2e6639ca03f4c2f270b23ed1a09b90413" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "566370012e4a1d32af2ef9035ff64d743cd81f36f25d2724e7b033e393b8247e" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7d499e40f572cfc29ecfb44f8376493df56a44213b1c1e9333b65499a0c288cd" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "53241e64de1e4470712b4061e2c82f44916d654bc532f8f1d12e5d5d4e136914" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fd168faab4546f988dc38fc56df2f71cf80c922e86d3f869be15a43f08328f99" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e578afe329abb0cba64802dfa480e00f2bbbb8c80be537791c24a31c853eb62f" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a3cb32dba55907ca907fc4f38f7ca05ef6db10a6af2dd1fa3c4db166e4ab9ffe" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7e4368ba8ec460b3c94de24ab0a04b6c799eb28df885cbbacfc3bb3ffa8c1e67" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m (x4 over 10m)    kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c4aaa8f8cd2dc1eff788baf04774c4ecc845568d00ed1b386df311ec224eb6f3" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Normal   SandboxChanged          56s (x551 over 10m)  kubelet            Pod sandbox changed, it will be killed and re-created.



azureuser@k8-master:~$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
default                demo-6c59fb8f77-2jq6k                        0/1     ContainerCreating   0          5m23s
kube-system            coredns-f9fd979d6-q8s9b                      1/1     Running             2          27h
kube-system            coredns-f9fd979d6-qnm4j                      1/1     Running             2          27h
kube-system            etcd-k8-master                               1/1     Running             2          27h
kube-system            kube-apiserver-k8-master                     1/1     Running             3          27h
kube-system            kube-controller-manager-k8-master            1/1     Running             3          27h
kube-system            kube-flannel-ds-kqz4t                        0/1     CrashLoopBackOff    92         27h
kube-system            kube-flannel-ds-szqzn                        1/1     Running             3          27h
kube-system            kube-flannel-ds-v9q47                        0/1     CrashLoopBackOff    142        27h
kube-system            kube-proxy-4mb47                             1/1     Running             2          27h
kube-system            kube-proxy-54m9b                             1/1     Running             2          27h
kube-system            kube-proxy-wdxfz                             1/1     Running             1          27h
kube-system            kube-scheduler-k8-master                     1/1     Running             3          27h
kubernetes-dashboard   dashboard-metrics-scraper-7b59f7d4df-zmlvs   0/1     ContainerCreating   0          27h
kubernetes-dashboard   kubernetes-dashboard-665f4c5ff-cnsvn         0/1     ContainerCreating   0          6h3m

To fix the flannel crashloopbackoff we did Kubeadm reset and after some time this problem showed up again.

Current we are working with one master and two worker node.

My cluster details as follows: azureuser@k8-master:~$ kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://52.150.11.168:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED

Docker version:

azureuser@k8-master:~$ sudo docker version
[sudo] password for azureuser: 
Client:
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        369ce74a3c
 Built:             Wed Oct 14 19:00:27 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       369ce74a3c
  Built:            Wed Oct 14 16:52:50 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu1~18.04.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

kubeadm version :

azureuser@k8-master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:15:05Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

The flannel is crashing whenever I tried to schedule pod creation.

-- Giridharan M
kubeadm
kubectl
kubernetes

1 Answer

12/29/2020

Background

I think your issue is cased by your 2 Flannel CNI pods CrashLoopBackOff status.

Your error

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory

is pointing that pod cannot be created due to lack of /run/flannel/subnet.env file. In Flannel Github document you can find:

Flannel runs a small, single binary agent called flanneld on each host, and is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space.

Meaning, to proper work, Flannel pod should be running on each node as it contains subnets information. From your outputs I can see that only 1 is working properly out of 3 Flannel pods.

NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
...
kube-system            kube-flannel-ds-kqz4t                        0/1     CrashLoopBackOff    92         27h
kube-system            kube-flannel-ds-szqzn                        1/1     Running             3          27h
kube-system            kube-flannel-ds-v9q47                        0/1     CrashLoopBackOff    142        27h

If mentioned pod was scheduled on node where flannel pod is not working it won't be created due to CNI network issues. Besides your demo pod, also kubernetes-dashboard pods have the same issue with ContainerCreating status.

Conclusion

Your demo pod cannot be scheduled as Kubernetes encounter some network issues related with flannel configuration file (...network: open /run/flannel/subnet.env: no such file or directory).

Your flannel pods restarts counts is very high as for 27 hours. You have to determine why and fix it. It might be lack of resources, network issues with your infrastructure or many other reasons. Once all flannel pods will be working correctly, your shouldn't encounter this error.

Solution

You have to make flannel pods works correctly on each node.

Additional Troubleshooting Details

For detailed investigation please provide

$ kubectl describe kube-flannel-ds-kqz4t -n kube-system
$ kubectl describe kube-flannel-ds-v9q47 -n kube-system

Logs details would be also helpful

$ kubectl logs kube-flannel-ds-kqz4t -n kube-system
$ kubectl logs kube-flannel-ds-v9q47 -n kube-system

Please replace kubectl get pods --all-namespaces with kubectl get pods -o wide -A and output of kubectl get nodes -o wide.

If you will provide those information, it should be possible to determine root cause of flannel pods issues and I will edit this answer with exact solution.

-- PjoterS
Source: StackOverflow