I am using Kubernetes using minikube and remote docker container. I am just starting so I deployed an application using a replica set and 10 replicas and defined a service for it.
Everything works fine, I am able to access the application through the service and it routes to node port. I can also check the status of replica and it says 10 nodes running.
E:\k8s>kubectl get rs web-rs
NAME DESIRED CURRENT READY AGE
web-rs 10 10 10 5h
However, in the logs, I see an error message as:
Nov 19 03:05:53 minikube kubelet[2737]: I1119 03:05:53.979651 2737 kubelet.go:1794] skipping pod synchronization - [container runtime is down]
Nov 19 03:52:10 minikube kubelet[2737]: I1119 03:52:10.024647 2737 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 8m1.440052328s ago; threshold is 3m0s]
Nov 19 03:52:10 minikube kubelet[2737]: I1119 03:52:10.265727 2737 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 8m1.681197091s ago; threshold is 3m0s]
Nov 19 03:52:10 minikube kubelet[2737]: I1119 03:52:10.608070 2737 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 8m2.023574099s ago; threshold is 3m0s]
Nov 19 03:52:11 minikube kubelet[2737]: I1119 03:52:11.140992 2737 kubelet.go:1794] skipping pod synchronization - [PLEG is not healthy: pleg was last seen active 8m1.519168715s ago; threshold is 3m0s]
I have checked the kubernetes error : skipping pod synchronization
docker is up. I can do kubectl get nodes and even run docker commands without any issues. On docker ps i do see many containers are in running state so it doesnt seem an issue with docker per se to me. ( I am not an expert on this and so may be wrong) Output of commands :
E:\k8s>kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready master 4d v1.10.0
E:\k8s>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6210dec0804c nigelpoulton/k8sbook "node ./app.js" 25 hours ago Up 25 hours k8s_hello-ctr_web-rs-5ts8s_default_8211e219-eb7f-11e8-aea3-080027545ad3_0
ba2cd1b5bf0f nigelpoulton/k8sbook "node ./app.js" 25 hours ago Up 25 hours k8s_hello-ctr_web-rs-74kvs_default_82094866-eb7f-11e8-aea3-080027545ad3_0
2d2195ef53da nigelpoulton/k8sbook "node ./app.js" 25 hours ago Up 25 hours k8s_hello-ctr_web-rs-pbzbs_default_821068db-eb7f-11e8-aea3-080027545ad3_0
fc344ab1dc4b nigelpoulton/k8sbook "node ./app.js" 25 hours ago Up 25 hours k8s_hello-ctr_web-rs-wxntx_default_8205c6a0-eb7f-11e8-aea3-080027545ad3_0
3cc60a78b04f nigelpoulton/k8sbook "node ./app.js" 25 hours ago Up 25 hours k8s_hello-ctr_web-rs-8gmx2_default_82094590-eb7f-11e8-aea3-080027545ad3_0
cf1aec9beac3 k8s.gcr.io/pause-amd64:3.1 "/pause" 25 hours ago Up 25 hours k8s_POD_web-rs-5ts8s_default_8211e219-eb7f-11e8-aea3-080027545ad3_0
b509bdecffbb k8s.gcr.io/pause-amd64:3.1 "/pause" 25 hours ago Up 25 hours k8s_POD_web-rs-74kvs_default_82094866-eb7f-11e8-aea3-080027545ad3_0
b23c46b84834 k8s.gcr.io/pause-amd64:3.1 "/pause" 25 hours ago Up 25 hours k8s_POD_web-rs-pbzbs_default_821068db-eb7f-11e8-aea3-080027545ad3_0
d79178bf194c k8s.gcr.io/pause-amd64:3.1 "/pause" 25 hours ago Up 25 hours k8s_POD_web-rs-wxntx_default_8205c6a0-eb7f-11e8-aea3-080027545ad3_0
e8d146389e12 k8s.gcr.io/pause-amd64:3.1 "/pause" 25 hours ago Up 25 hours k8s_POD_web-rs-8gmx2_default_82094590-eb7f-11e8-aea3-080027545ad3_0
0dbad5b7f578 nigelpoulton/k8sbook "node ./app.js" 26 hours ago Up 26 hours k8s_hello-ctr_web-rs-7l4hm_default_b8020125-eb7d-11e8-aea3-080027545ad3_0
ad299c1714be nigelpoulton/k8sbook "node ./app.js" 26 hours ago Up 26 hours k8s_hello-ctr_web-rs-72mfk_default_b80192f4-eb7d-11e8-aea3-080027545ad3_0
f17a60d958f4 nigelpoulton/k8sbook "node ./app.js" 26 hours ago Up 26 hours k8s_hello-ctr_web-rs-dcd7r_default_b7f5fbcf-eb7d-11e8-aea3-080027545ad3_0
5f70a24cc6cb nigelpoulton/k8sbook "node ./app.js" 26 hours ago Up 26 hours k8s_hello-ctr_web-rs-64w2l_default_b7fab867-eb7d-11e8-aea3-080027545ad3_0
f428779757b4 nigelpoulton/k8sbook "node ./app.js" 26 hours ago Up 26 hours k8s_hello-ctr_web-rs-hg47s_default_b7fc5fb1-eb7d-11e8-aea3-080027545ad3_0
22b39107aaf1 k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_web-rs-72mfk_default_b80192f4-eb7d-11e8-aea3-080027545ad3_0
d54c134dc152 k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_web-rs-7l4hm_default_b8020125-eb7d-11e8-aea3-080027545ad3_0
bbf6fc9776d8 k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_web-rs-dcd7r_default_b7f5fbcf-eb7d-11e8-aea3-080027545ad3_0
d07a69b55ffe k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_web-rs-hg47s_default_b7fc5fb1-eb7d-11e8-aea3-080027545ad3_0
8a2617bb9603 k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_web-rs-64w2l_default_b7fab867-eb7d-11e8-aea3-080027545ad3_0
69fde6763598 4689081edb10 "/storage-provisioner" 26 hours ago Up 26 hours k8s_storage-provisioner_storage-provisioner_kube-system_7728aa28-e884-11e8-ad06-080027545ad3_3
0d7c25eb5bb2 80cc5ea4b547 "/kube-dns --domain=…" 26 hours ago Up 26 hours k8s_kubedns_kube-dns-86f4d74b45-xp4q6_kube-system_74236c52-e884-11e8-ad06-080027545ad3_2
f31cdf99455c 0dab2435c100 "/dashboard --insecu…" 26 hours ago Up 26 hours k8s_kubernetes-dashboard_kubernetes-dashboard-6f4cfc5d87-h4v4c_kube-system_764a66ba-e884-11e8-ad06-080027545ad3_2
e943abe86f72 bfc21aadc7d3 "/usr/local/bin/kube…" 26 hours ago Up 26 hours k8s_kube-proxy_kube-proxy-6pzss_kube-system_28b9825b-eb7c-11e8-aea3-080027545ad3_0
e5f65cd0324b k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_kube-proxy-6pzss_kube-system_28b9825b-eb7c-11e8-aea3-080027545ad3_0
5c912fef63f3 nigelpoulton/k8sbook "node ./app.js" 26 hours ago Up 26 hours k8s_hello-ctr_hello-pod_default_5490f219-e887-11e8-ad06-080027545ad3_1
8caf216c046d 367cdc8433a4 "/coredns -conf /etc…" 26 hours ago Up 26 hours k8s_coredns_coredns-c4cffd6dc-fcnmn_kube-system_76d2d6dd-e884-11e8-ad06-080027545ad3_1
f829c9f5ee2b 6f7f2dc7fab5 "/sidecar --v=2 --lo…" 26 hours ago Up 26 hours k8s_sidecar_kube-dns-86f4d74b45-xp4q6_kube-system_74236c52-e884-11e8-ad06-080027545ad3_1
12a8246e191b k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_hello-pod_default_5490f219-e887-11e8-ad06-080027545ad3_1
13cbc566725d k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_coredns-c4cffd6dc-fcnmn_kube-system_76d2d6dd-e884-11e8-ad06-080027545ad3_1
6c3c993f5fec k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_storage-provisioner_kube-system_7728aa28-e884-11e8-ad06-080027545ad3_1
0c2b14a7a754 c2ce1ffb51ed "/dnsmasq-nanny -v=2…" 26 hours ago Up 26 hours k8s_dnsmasq_kube-dns-86f4d74b45-xp4q6_kube-system_74236c52-e884-11e8-ad06-080027545ad3_1
8943ff904243 k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_kubernetes-dashboard-6f4cfc5d87-h4v4c_kube-system_764a66ba-e884-11e8-ad06-080027545ad3_1
be4952f694d1 k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_kube-dns-86f4d74b45-xp4q6_kube-system_74236c52-e884-11e8-ad06-080027545ad3_1
4a191bb8c1a2 52920ad46f5b "etcd --data-dir=/da…" 26 hours ago Up 26 hours k8s_etcd_etcd-minikube_kube-system_b3f4c74d209e4c402cf5e20b463a8d85_0
655e9358f8ef ad86dbed1555 "kube-controller-man…" 26 hours ago Up 26 hours k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_afcd561c923daaf28e9e843f2e2ea926_0
78f37e068a4d 704ba848e69a "kube-scheduler --ad…" 26 hours ago Up 26 hours k8s_kube-scheduler_kube-scheduler-minikube_kube-system_31cf0ccbee286239d451edb6fb511513_0
12d37bb08006 9c16409588eb "/opt/kube-addons.sh" 26 hours ago Up 26 hours k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_8b52f08746ac78d32737b5f7fdffec52_1
5b77f09504cd af20925d51a3 "kube-apiserver --ad…" 26 hours ago Up 26 hours k8s_kube-apiserver_kube-apiserver-minikube_kube-system_f45680c13ebafc8e721762e2f2801457_0
3cced8f41bae k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_kube-scheduler-minikube_kube-system_31cf0ccbee286239d451edb6fb511513_0
a04ff8f69912 k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_kube-controller-manager-minikube_kube-system_afcd561c923daaf28e9e843f2e2ea926_0
398cc849dcf7 k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_etcd-minikube_kube-system_b3f4c74d209e4c402cf5e20b463a8d85_0
11c67dd15c11 k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_kube-apiserver-minikube_kube-system_f45680c13ebafc8e721762e2f2801457_0
5f2910171b4a k8s.gcr.io/pause-amd64:3.1 "/pause" 26 hours ago Up 26 hours k8s_POD_kube-addon-manager-minikube_kube-system_8b52f08746ac78d32737b5f7fdffec52_1
I am not able to ascertain the cause of this error and how to debug this. Could someone please advise?
Regards