Do I need drain the node to upgrade the kubelet version?

5/28/2018

I'm planning to upgrade kubernetes cluster version from v1.7.10 to v1.8.12. but as this issue state that all containers will be restart because of the spec hash change. So, what's the suggested upgrade procedure ? do I need to drain the node before upgrading kubelet version, or just do an in-place upgrade and let all the containers restarting ? what's the difference ?

Also, since upgrading to v1.9.0 will also cause containers restarting, can I upgrade v1.7.10 directly to v1.10.3 ? In this way I can avoid two time-consuming upgrades to v1.8 and v1.9 at least. Is there any constraints provent me from doing this ?

Any suggestion will be appreciated.

-- Kun Li
kubernetes

3 Answers

5/28/2018

It is always a good idea to drain/cordon a node while you do operational tasks like upgrades. Once the upgrade is completed, uncordon the node, and then repeat the process for the other nodes in the cluster.

Re: skipping versions, as far as I know there are no constraints against skipping minor versions in an upgrade.

-- Murcurio
Source: StackOverflow

5/31/2018

After some testing and research, I come to some conclusion:

  1. Drain the node is not a must. At least, drain can't evict daemonsets. But drain a node is recommend way to upgrade kubernetes, since this can at most extent reduce the impact to applications that's deployed using deployment.

    Also, sometimes, drain a node is neccesary. For instance, from kubernetes v1.10, the log files of all pods have changed from /var/log/pods/pod-id/container_id.log to /var/log/pods/pod-id/container/id.log, so when upgrade to v1.10, all pods have to restart to use the new log file, otherwise, you can't access their logs through 'kubectl logs' command. At this time, drain a node is helpful, and to those pods that can't be evicted, like daemonsets, we have to restart them manually.

  2. Skip minor version upgrading is not supported, especially in a HA setup, with multiple masters, which can be proved somehow by GKE's supported upgrades policy. Also, skip minor version upgrading implies some risk sometimes.

    Again, as an example, upgrade to v1.10. In 1.10, objects in the apps API group began persisting in etcd in apps/v1 format, which can be handled very well by v1.9, but v1.8 can't. So, when you upgrade kubernetes from v1.8 to 1.10, in a HA setup, some masters have been upgraded, some haven't, will bring some weird problems, like deployment/daemonset can't be handled properly, more info refer to my another question. So, such upgrade should be avoid as long as possible.

-- Kun Li
Source: StackOverflow

5/28/2018

The Kubeadm cluster upgrade page does not recommend skipping a major version: 17 to 1.8, then 1.8 to 1.9 is still the recommended path.
I guess it is the same idea for kubernetes itself upgrades.

do I need to drain the node before upgrading kubelet version

That is what the documentation recommends:

For each host (referred to as $HOST below) in your cluster, upgrade kubelet by executing the following commands:

Prepare the host for maintenance, marking it unschedulable and evicting the workload:

kubectl drain $HOST --ignore-daemonsets
-- VonC
Source: StackOverflow