minimise disruption on weave network upgrade on kubernetes

10/8/2019

I would like to upgrade my weave network from version 2.5.0 to 2.5.2. I understand that it's "as simple" as updating the weave daemonset.... however, i was wondering if there is a way that this can be done with minimal disruption to running pods on the system.

An simple example in my mind would be to:

  • cordon node1
  • drain node1 of all pods
  • update weave on node1
  • uncordon node1

... then rinse and repeat for each k8s node until all done.

-- yee379
cni
kubernetes
networking
upgrade
weave

1 Answer

10/9/2019

Basing on the weave net documentation

Upgrading the Daemon Sets

The DaemonSet definition specifies Rolling Updates, so when you apply a new version Kubernetes will automatically restart the Weave Net pods one by one.

With RollingUpdate update strategy, after you update a DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods will be created automatically, in a controlled fashion.

As i could read in another stackoverflow answer

It is possible to perform rolling updates with no downtime using a DeamonSet as of Today! What you need is to have at least 2 nodes running on your cluster and set maxUnavailable to 1 in your DaemonSet configuration.

Assuming the previous configuration, when an update is pushed, a first node will start updating. The second will waiting until the first completes. Upon success, the second does the same.

The major drawback is that you need to keep 2 nodes runnings continuously or to take actions to spawn/kill a node before/after an update.

So i think the best option for you to upgrade your CNI plugin is using DaemonSet with rolling update and set maxUnavailable to 1 in your DaemonSet configuration.

-- jt97
Source: StackOverflow