Longhorn PVC redistribution volumes on all nodes

8/26/2020

We are using 3 master - 4 worker setup cluster. Recently due to disk pressure we had to add another worker to our cluster and we would like to redistribute some of the existing persistent volume claims to new worker.

Right now,

2 out of 4 Longhorn nodes are not schedulable due to not enough disk space. I can cordon 2 of these nonschedulable nodes and manually delete some of the pvcs on these cordon stated nodes in order to re-create them on new worker. But I was wondering if there is another way to automate this process. I think you can understand better my question and what I'm trying to do with the provided image. Thanks for your answers already!

Longhorn version : 0.8 enter image description here

-- Çağdaş Özgür
automation
cluster-computing
kubernetes
persistent-volumes
rancher

1 Answer

11/5/2020

I think it’s better for you to just create a new replica. Instead of deleting your pv/pvc. Click on the longhorn volume that you want to move around to a different node (using longhorn dashboard, in volume menu).

In the context menu there should be option to Update Replicas Count if your volume is attached.

Increase the replica so longhorn create more replica in the healthy node. Wait for it to finish rebuilding. Let’s say previously you have 2 replicas, now you have 3 (one new replica in the healthy node). Reduce the replica count again to 2. Then delete your replica in the node you want to cordon.

So this way:

  • No need to delete your PV/PVC then reattach
  • Zero downtime

Also, longhorn has reached version 1 by now. It’s a good idea to upgrade.

-- lucernae
Source: StackOverflow