Unable to delete the apache solr operator CRD in Kubernetes

9/5/2021

Here's something interesting I am trying to figure out and looking for some help.

I am using following version

Client Version: v1.21.2
Server Version: v1.20.7

I had installed the apache solr operator using

https://solr.apache.org/operator/downloads/crds/v0.3.0/all-with-dependencies.yaml

Due to some issue I tried deleting it by removing the namespace itself. (Never do this!!)

Now I see that namespace is deleted but the custom resources are not getting deleted.

kubectl get --show-kind solrclouds.solr.apache.org -n solr-wcnp-poc --v=6
I0905 14:21:39.593361   39140 round_trippers.go:454] GET https://oidc.us-dev-aks-ad-solrpoc.k8s.us.abcd.net/apis/solr.apache.org/v1beta1/namespaces/solr-wcnp-poc/solrclouds?limit=500 200 OK in 292 milliseconds
NAME                                VERSION   TARGETVERSION   DESIREDNODES   NODES   READYNODES   UPTODATENODES   AGE
solrcloud.solr.apache.org/explore   8.8.2                     1              1       0            1               7d19h

I too a help of the almighty google search and figured out that patching finalizers does the trick. So I tried the same but now seeing something interesting!

kubectl patch solrcloud.solr.apache.org/explore -p '{"metadata":{"finalizers":[]}}' -n solr-wcnp-poc --type=merge --v=6

I0905 14:39:00.028568   55762 round_trippers.go:454] GET https://oidc.us-dev-aks-ad-solrpoc.k8s.us.abcd.net/apis/solr.apache.org/v1beta1/namespaces/solr-wcnp-poc/solrclouds/explore 200 OK in 343 milliseconds
I0905 14:39:00.335421   55762 round_trippers.go:454] PATCH https://oidc.us-dev-aks-ad-solrpoc.k8s.us.abcd.net/apis/solr.apache.org/v1beta1/namespaces/solr-wcnp-poc/solrclouds/explore?fieldManager=kubectl-patch 404 Not Found in 303 milliseconds
I0905 14:39:00.338733   55762 helpers.go:216] server response object: [{  "kind": "Status",  "apiVersion": "v1",  "metadata": {},  "status": "Failure",  "message": "namespaces \"solr-wcnp-poc\" not found",  "reason": "NotFound",  "details": {    "name": "solr-wcnp-poc",    "kind": "namespaces"  },  "code": 404}]
F0905 14:39:00.339471   55762 helpers.go:115] Error from server (NotFound): namespaces "solr-wcnp-poc" not found

My question is, If the kubectl patch command is not able to locate the namespace then how come the kubectl get --showkind is showing the resources under the deleted namespace?

Can anybody please help me understand this?

NOTE: Now the k8s cluster is not letting me either delete the Solr-operator (the kubectl delete hangs indefinitely) like

customresourcedefinition.apiextensions.k8s.io "solrbackups.solr.apache.org" deleted
customresourcedefinition.apiextensions.k8s.io "solrclouds.solr.apache.org" deleted
customresourcedefinition.apiextensions.k8s.io "solrprometheusexporters.solr.apache.org" deleted
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io "zookeeperclusters.zookeeper.pravega.io" deleted

^C

or create a new operator (kubectl says

customresourcedefinition.apiextensions.k8s.io/solrbackups.solr.apache.org created
customresourcedefinition.apiextensions.k8s.io/solrprometheusexporters.solr.apache.org created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Error from server (AlreadyExists): error when creating "solr_operator.yaml": object is being deleted: customresourcedefinitions.apiextensions.k8s.io "solrclouds.solr.apache.org" already exists
Error from server (AlreadyExists): error when creating "solr_operator.yaml": object is being deleted: customresourcedefinitions.apiextensions.k8s.io "zookeeperclusters.zookeeper.pravega.io" already exists
-- happydude
kubectl
kubernetes
solr

1 Answer

10/7/2021

I had the same issue and manage to delete the custom resource by recreating the namespace and run

kubectl patch solrcloud.solr.apache.org <NAME> -p '{"metadata":{"finalizers":[]}}' -n <RESTORED_NAMESPACE> --type=merge --v=6

The same solution worked with the custom resources, perconaxtradbclusters.pxc.percona.com, and zookeeperclusters.zookeeper.pravega.io that I was not able to deleteĀ 

-- Maoz Zadok
Source: StackOverflow