kubernetes cluster is down after updating image on master node

11/28/2018

Team, any hint on how can i fix this? Entire cluster went down after upgrading master image.

Logs state:

10.3.0.9-37d77862 docker[5521]: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Nov 28 02:36:08 10.3.0.9-37d77862 docker[5521]: W1128 10:36:08.271939    5597 kube_boot.go:169] error applying channel "s3://test-bucket
Nov 28 02:37:08 10.3.0.9-37d77862 docker[5521]: I1128 10:37:08.272384    5597 vol.go:81] Found volumes: [{"ID":"01","LocalDevice":"dummy_dev","AttachedTo":"node_ip","M
Nov 28 02:37:08 10.3.0.9-37d77862 docker[5521]: I1128 10:37:08.272473    5597 kube_boot.go:132] Found etcd cluster spec on volume "01": {"clusterKey":"main","nodeName":"a","nNov 28 02:37:08 10.3.0.9-37d77862 docker[5521]: I1128 10:37:08.272509    5597 etcd_cluster.go:182] EtcdJoinExistingCluster enabled.
Nov 28 02:37:08 10.3.0.9-37d77862 docker[5521]: I1128 10:37:08.272513    5597 etcd_cluster.go:184] Setting etcd init cluster state to existing ...
Nov 28 02:37:08 10.3.0.9-37d77862 docker[5521]: 2018-11-28 10:37:08.273030 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Nov 28 02:37:08 10.3.0.9-37d77862 docker[5521]: I1128 10:37:08.273052    5597 etcd_cluster.go:424] Checking if etcd member at https://127.0.0.1:4001/healthz is already runnin
Nov 28 02:37:08 10.3.0.9-37d77862 docker[5521]: I1128 10:37:08.273306    5597 etcd_cluster.go:382] checkMemberRunning doer.RoundTrip failed, dial tcp 127.0.0.1:4001: getsocko
Nov 28 02:37:08 10.3.0.9-37d77862 docker[5521]: I1128 10:37:08.273328    5597 etcd_cluster.go:436] Local etcd isRunning = false
Nov 28 02:37:18 10.3.0.9-37d77862 docker[5521]: E1128 10:37:18.273737    5597 etcd_cluster.go:495] etcd MemberList failed: context deadline exceeded
Nov 28 02:37:18 10.3.0.9-37d77862 docker[5521]: E1128 10:37:18.273785    5597 etcd_cluster.go:518] findEtcdMember failed, err: context deadline exceeded, retry in 3
Nov 28 02:37:31 10.3.0.9-37d77862 docker[5521]: E1128 10:37:31.274348    5597 etcd_cluster.go:495] etcd MemberList failed: context deadline exceeded
Nov 28 02:37:31 10.3.0.9-37d77862 docker[5521]: E1128 10:37:31.274372    5597 etcd_cluster.go:518] findEtcdMember failed, err: context deadline exceeded, retry in 3
Nov 28 02:37:43 10.3.0.9-37d77862 docker[5521]: E1128 10:37:43.274280    5597 etcd_cluster.go:495] etcd MemberList failed: context deadline exceeded
Nov 28 02:37:43 10.3.0.9-37d77862 docker[5521]: E1128 10:37:43.274304    5597 etcd_cluster.go:518] findEtcdMember failed, err: context deadline exceeded, retry in 3
Nov 28 02:37:55 10.3.0.9-37d77862 docker[5521]: E1128 10:37:55.274509    5597 etcd_cluster.go:495] etcd MemberList failed: context deadline exceeded
Nov 28 02:37:55 10.3.0.9-37d77862 docker[5521]: E1128 10:37:55.274530    5597 etcd_cluster.go:518] findEtcdMember failed, err: context deadline exceeded, retry in 3
-- AhmFM
etcd
kube-apiserver
kubernetes

0 Answers