All pods get restarted while cluster auto upgrade is disabled

10/10/2018

The cluster disabled auto upgrade. While this morning (Oct. 10th, 2018, around 10:00am PST), nearly all pods restarted. Any possibility that it's relevant with K8s or GKE new version release? If not, what's the reason for the restarts?

-- Hai Shang
google-kubernetes-engine
kubernetes

2 Answers

10/11/2018

Without looking at the pod events and other logs, and by assuming the nodes restarted as well, I can speculate an emergency maintenance operation on the physical elements of the datacenter in which the nodes were running. This can be confirmed from Operations History and Stackdriver logs etc.

Also, look for node's exhausted resources (OOM killer) which cause pods to restart.

-- Asif Tanwir
Source: StackOverflow

10/11/2018
  • When you are disabling auto upgrade ensure that disabling this feature(auto upgrade) does not halt in-progress upgrades.
  • Did you disable auto-upgrade on all node pools in the cluster & did you check whether any upgrade is in progress?
-- manoj kumar
Source: StackOverflow