Kubernetes minikube error: skipping pod synchronization - [container runtime is down PLEG is not healthy]

11/19/2018

I am using Kubernetes using minikube and remote docker container. I am just starting so I deployed an application using a replica set and 10 replicas and defined a service for it.

Everything works fine, I am able to access the application through the service and it routes to node port. I can also check the status of replica and it says 10 nodes running.

E:\k8s>kubectl get rs web-rs
NAME     DESIRED   CURRENT   READY   AGE
web-rs   10        10        10      5h

However, in the logs, I see an error message as:

    Nov 19 03:05:53 minikube kubelet[2737]: I1119 03:05:53.979651    2737 kubelet.go:1794] skipping pod synchronization - [container runtime is down]
Nov 19 03:52:10 minikube kubelet[2737]: I1119 03:52:10.024647    2737 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 8m1.440052328s ago; threshold is 3m0s]
Nov 19 03:52:10 minikube kubelet[2737]: I1119 03:52:10.265727    2737 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 8m1.681197091s ago; threshold is 3m0s]
Nov 19 03:52:10 minikube kubelet[2737]: I1119 03:52:10.608070    2737 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 8m2.023574099s ago; threshold is 3m0s]
Nov 19 03:52:11 minikube kubelet[2737]: I1119 03:52:11.140992    2737 kubelet.go:1794] skipping pod synchronization - [PLEG is not healthy: pleg was last seen active 8m1.519168715s ago; threshold is 3m0s]

I have checked the kubernetes error : skipping pod synchronization

docker is up. I can do kubectl get nodes and even run docker commands without any issues. On docker ps i do see many containers are in running state so it doesnt seem an issue with docker per se to me. ( I am not an expert on this and so may be wrong) Output of commands :

E:\k8s>kubectl get nodes
NAME       STATUS   ROLES    AGE   VERSION
minikube   Ready    master   4d    v1.10.0

E:\k8s>docker ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS               NAMES
6210dec0804c        nigelpoulton/k8sbook         "node ./app.js"          25 hours ago        Up 25 hours                             k8s_hello-ctr_web-rs-5ts8s_default_8211e219-eb7f-11e8-aea3-080027545ad3_0
ba2cd1b5bf0f        nigelpoulton/k8sbook         "node ./app.js"          25 hours ago        Up 25 hours                             k8s_hello-ctr_web-rs-74kvs_default_82094866-eb7f-11e8-aea3-080027545ad3_0
2d2195ef53da        nigelpoulton/k8sbook         "node ./app.js"          25 hours ago        Up 25 hours                             k8s_hello-ctr_web-rs-pbzbs_default_821068db-eb7f-11e8-aea3-080027545ad3_0
fc344ab1dc4b        nigelpoulton/k8sbook         "node ./app.js"          25 hours ago        Up 25 hours                             k8s_hello-ctr_web-rs-wxntx_default_8205c6a0-eb7f-11e8-aea3-080027545ad3_0
3cc60a78b04f        nigelpoulton/k8sbook         "node ./app.js"          25 hours ago        Up 25 hours                             k8s_hello-ctr_web-rs-8gmx2_default_82094590-eb7f-11e8-aea3-080027545ad3_0
cf1aec9beac3        k8s.gcr.io/pause-amd64:3.1   "/pause"                 25 hours ago        Up 25 hours                             k8s_POD_web-rs-5ts8s_default_8211e219-eb7f-11e8-aea3-080027545ad3_0
b509bdecffbb        k8s.gcr.io/pause-amd64:3.1   "/pause"                 25 hours ago        Up 25 hours                             k8s_POD_web-rs-74kvs_default_82094866-eb7f-11e8-aea3-080027545ad3_0
b23c46b84834        k8s.gcr.io/pause-amd64:3.1   "/pause"                 25 hours ago        Up 25 hours                             k8s_POD_web-rs-pbzbs_default_821068db-eb7f-11e8-aea3-080027545ad3_0
d79178bf194c        k8s.gcr.io/pause-amd64:3.1   "/pause"                 25 hours ago        Up 25 hours                             k8s_POD_web-rs-wxntx_default_8205c6a0-eb7f-11e8-aea3-080027545ad3_0
e8d146389e12        k8s.gcr.io/pause-amd64:3.1   "/pause"                 25 hours ago        Up 25 hours                             k8s_POD_web-rs-8gmx2_default_82094590-eb7f-11e8-aea3-080027545ad3_0
0dbad5b7f578        nigelpoulton/k8sbook         "node ./app.js"          26 hours ago        Up 26 hours                             k8s_hello-ctr_web-rs-7l4hm_default_b8020125-eb7d-11e8-aea3-080027545ad3_0
ad299c1714be        nigelpoulton/k8sbook         "node ./app.js"          26 hours ago        Up 26 hours                             k8s_hello-ctr_web-rs-72mfk_default_b80192f4-eb7d-11e8-aea3-080027545ad3_0
f17a60d958f4        nigelpoulton/k8sbook         "node ./app.js"          26 hours ago        Up 26 hours                             k8s_hello-ctr_web-rs-dcd7r_default_b7f5fbcf-eb7d-11e8-aea3-080027545ad3_0
5f70a24cc6cb        nigelpoulton/k8sbook         "node ./app.js"          26 hours ago        Up 26 hours                             k8s_hello-ctr_web-rs-64w2l_default_b7fab867-eb7d-11e8-aea3-080027545ad3_0
f428779757b4        nigelpoulton/k8sbook         "node ./app.js"          26 hours ago        Up 26 hours                             k8s_hello-ctr_web-rs-hg47s_default_b7fc5fb1-eb7d-11e8-aea3-080027545ad3_0
22b39107aaf1        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_web-rs-72mfk_default_b80192f4-eb7d-11e8-aea3-080027545ad3_0
d54c134dc152        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_web-rs-7l4hm_default_b8020125-eb7d-11e8-aea3-080027545ad3_0
bbf6fc9776d8        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_web-rs-dcd7r_default_b7f5fbcf-eb7d-11e8-aea3-080027545ad3_0
d07a69b55ffe        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_web-rs-hg47s_default_b7fc5fb1-eb7d-11e8-aea3-080027545ad3_0
8a2617bb9603        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_web-rs-64w2l_default_b7fab867-eb7d-11e8-aea3-080027545ad3_0
69fde6763598        4689081edb10                 "/storage-provisioner"   26 hours ago        Up 26 hours                             k8s_storage-provisioner_storage-provisioner_kube-system_7728aa28-e884-11e8-ad06-080027545ad3_3
0d7c25eb5bb2        80cc5ea4b547                 "/kube-dns --domain=…"   26 hours ago        Up 26 hours                             k8s_kubedns_kube-dns-86f4d74b45-xp4q6_kube-system_74236c52-e884-11e8-ad06-080027545ad3_2
f31cdf99455c        0dab2435c100                 "/dashboard --insecu…"   26 hours ago        Up 26 hours                             k8s_kubernetes-dashboard_kubernetes-dashboard-6f4cfc5d87-h4v4c_kube-system_764a66ba-e884-11e8-ad06-080027545ad3_2
e943abe86f72        bfc21aadc7d3                 "/usr/local/bin/kube…"   26 hours ago        Up 26 hours                             k8s_kube-proxy_kube-proxy-6pzss_kube-system_28b9825b-eb7c-11e8-aea3-080027545ad3_0
e5f65cd0324b        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_kube-proxy-6pzss_kube-system_28b9825b-eb7c-11e8-aea3-080027545ad3_0
5c912fef63f3        nigelpoulton/k8sbook         "node ./app.js"          26 hours ago        Up 26 hours                             k8s_hello-ctr_hello-pod_default_5490f219-e887-11e8-ad06-080027545ad3_1
8caf216c046d        367cdc8433a4                 "/coredns -conf /etc…"   26 hours ago        Up 26 hours                             k8s_coredns_coredns-c4cffd6dc-fcnmn_kube-system_76d2d6dd-e884-11e8-ad06-080027545ad3_1
f829c9f5ee2b        6f7f2dc7fab5                 "/sidecar --v=2 --lo…"   26 hours ago        Up 26 hours                             k8s_sidecar_kube-dns-86f4d74b45-xp4q6_kube-system_74236c52-e884-11e8-ad06-080027545ad3_1
12a8246e191b        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_hello-pod_default_5490f219-e887-11e8-ad06-080027545ad3_1
13cbc566725d        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_coredns-c4cffd6dc-fcnmn_kube-system_76d2d6dd-e884-11e8-ad06-080027545ad3_1
6c3c993f5fec        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_storage-provisioner_kube-system_7728aa28-e884-11e8-ad06-080027545ad3_1
0c2b14a7a754        c2ce1ffb51ed                 "/dnsmasq-nanny -v=2…"   26 hours ago        Up 26 hours                             k8s_dnsmasq_kube-dns-86f4d74b45-xp4q6_kube-system_74236c52-e884-11e8-ad06-080027545ad3_1
8943ff904243        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_kubernetes-dashboard-6f4cfc5d87-h4v4c_kube-system_764a66ba-e884-11e8-ad06-080027545ad3_1
be4952f694d1        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_kube-dns-86f4d74b45-xp4q6_kube-system_74236c52-e884-11e8-ad06-080027545ad3_1
4a191bb8c1a2        52920ad46f5b                 "etcd --data-dir=/da…"   26 hours ago        Up 26 hours                             k8s_etcd_etcd-minikube_kube-system_b3f4c74d209e4c402cf5e20b463a8d85_0
655e9358f8ef        ad86dbed1555                 "kube-controller-man…"   26 hours ago        Up 26 hours                             k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_afcd561c923daaf28e9e843f2e2ea926_0
78f37e068a4d        704ba848e69a                 "kube-scheduler --ad…"   26 hours ago        Up 26 hours                             k8s_kube-scheduler_kube-scheduler-minikube_kube-system_31cf0ccbee286239d451edb6fb511513_0
12d37bb08006        9c16409588eb                 "/opt/kube-addons.sh"    26 hours ago        Up 26 hours                             k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_8b52f08746ac78d32737b5f7fdffec52_1
5b77f09504cd        af20925d51a3                 "kube-apiserver --ad…"   26 hours ago        Up 26 hours                             k8s_kube-apiserver_kube-apiserver-minikube_kube-system_f45680c13ebafc8e721762e2f2801457_0
3cced8f41bae        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_kube-scheduler-minikube_kube-system_31cf0ccbee286239d451edb6fb511513_0
a04ff8f69912        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_kube-controller-manager-minikube_kube-system_afcd561c923daaf28e9e843f2e2ea926_0
398cc849dcf7        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_etcd-minikube_kube-system_b3f4c74d209e4c402cf5e20b463a8d85_0
11c67dd15c11        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_kube-apiserver-minikube_kube-system_f45680c13ebafc8e721762e2f2801457_0
5f2910171b4a        k8s.gcr.io/pause-amd64:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_kube-addon-manager-minikube_kube-system_8b52f08746ac78d32737b5f7fdffec52_1

I am not able to ascertain the cause of this error and how to debug this. Could someone please advise?

Regards

-- varkashy
docker
kubelet
kubernetes
minikube

0 Answers