K
Q

Unable to delete perconaxtradbclusters which are in errored state

4/21/2020

I've installed Percona XtraDB on kubernetes using 1.3.0 operator.

After using it, I wanted to delete the namespace. So I deleted them in the order which I applied them. Everything is deleted and nothing is visible in svc, pods but there are two resources which are in errored state and cannot be deleted.

~ kubectl get perconaxtradbclusters -n pxc
NAME       ENDPOINT   STATUS   PXC   PROXYSQL   AGE
cluster1              Error    0     0          4h1m
cluster2              Error    0     0          3h34m

I am not able to delete both of them and due to this I'm not able to create a cluster with the same name.

When I run the delete command, it gets stuck forever

~ kubectl delete  perconaxtradbclusters -n pxc cluster1
perconaxtradbcluster.pxc.percona.com "cluster1" deleted

The command execution is never completed.

The yaml of the object

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"pxc.percona.com/v1-3-0","kind":"PerconaXtraDBCluster"}
  creationTimestamp: "2020-04-21T18:06:13Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2020-04-21T18:38:33Z"
  finalizers:
  - delete-pxc-pods-in-order
  generation: 2
  name: cluster2
  namespace: pxc
  resourceVersion: "5445879"
  selfLink: /apis/pxc.percona.com/v1/namespaces/pxc/perconaxtradbclusters/cluster2
  uid: 8c100840-b7a8-40d1-b976-1f80c469622b
spec:
  allowUnsafeConfigurations: false
  backup:
    image: percona/percona-xtradb-cluster-operator:1.3.0-backup
    schedule:
    - keep: 3
      name: sat-night-backup
      schedule: 0 0 * * 6
      storageName: s3-us-west
    - keep: 5
      name: daily-backup
      schedule: 0 0 * * *
      storageName: fs-pvc
    serviceAccountName: percona-xtradb-cluster-operator
    storages:
      fs-pvc:
        type: filesystem
        volume:
          persistentVolumeClaim:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 6Gi
      s3-us-west:
        s3:
          bucket: S3-BACKUP-BUCKET-NAME-HERE
          credentialsSecret: my-cluster-name-backup-s3
          region: us-west-2
        type: s3
  pmm:
    enabled: false
    image: percona/percona-xtradb-cluster-operator:1.3.0-pmm
    serverHost: monitoring-service
    serverUser: pmm
  proxysql:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    enabled: true
    gracePeriod: 30
    image: percona/percona-xtradb-cluster-operator:1.3.0-proxysql
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      requests:
        cpu: 600m
        memory: 1G
    size: 3
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 2Gi
  pxc:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    gracePeriod: 600
    image: percona/percona-xtradb-cluster-operator:1.3.0-pxc
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      requests:
        cpu: 600m
        memory: 4G
    size: 3
    volumeSpec:
      persistentVolumeClaim:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 60Gi
        storageClassName: local-storage
  secretsName: my-cluster-secrets
  sslInternalSecretName: my-cluster-ssl-internal
  sslSecretName: my-cluster-ssl
status:
  conditions:
  - lastTransitionTime: "2020-04-21T18:06:13Z"
    message: 'wrong PXC options: set version: new version: Malformed version: '
    reason: ErrorReconcile
    status: "True"            
    type: Error
  message:
  - 'Error: wrong PXC options: set version: new version: Malformed version: '
  proxysql:
    ready: 0
  pxc:
    ready: 0
  state: Error

How can I get rid of them

-- Pardha
kubernetes
percona-xtradb-cluster

Similar Questions

Kubernetes Istio latency path wise in Grafana
Proper dockerfile for ansible which runs on k8s cluster
Kuberentes-Java-Client equivalent of getting all objects matching label
Kubernetes: unable to join a remote master node

1 Answer

4/21/2020

Your perconaxtradbclusters yaml example mentions pvc resources, so you'll probably have to delete the associated pvc first, if you haven't already done so.

Can you edit the resources to remove the finalizer blocks, and try delete them again?

kubectl edit perconaxtradbclusters cluster1 -n pxc

and delete

finalizers: 
- delete-pxc-pods-in-order

If there's nothing left relying on those resources, that is.

Edit:

I would generally only use this method if I've exhausted all other possibilities and I can't find the hanging resources that are blocking the deletion. I did some digging around. This comment here describes other steps to take before resorting to removing the finalizers. - Check that the API services are available - Find any lingering resources that still exist and delete them.

-- Ruairios
Source: StackOverflow