Data on block storage is all gone after kubeadm upgrade


I personally never experienced this kind of issue before, but today, after our kubeadm based on Kubernetes Cluster upgrade (v1.11.2 -> v1.11.5), one of our block storage's data is all gone (Postgres DB).

I even don't know this is k8s or provider (DigitalOcean) issues also there's no clue to start investigating.

Other block storages are all fine. Data is all persisted.

Our environment :

  • 1 master & 3 worker Kubernetes Cluster based on kubeadm.
  • Running on DigitalOcean (Using DigitalOcean CSI driver v0.2.0.)

I basically followed this document from Kubernetes.

Drained all Pods on every node, upgrade kubelet and kubeadm to v1.11.5, then kubeadm upgrade apply v1.11.5 on the Master.

Did anyone experience this issue before? Did I do something wrong in my step? Or at least any hits to start to investigate why this thing happened?

I feel like a blind now. Need your help badly.


-- Yechan Kim

0 Answers