Consul was uninstalled from k3s cluster yet it is still trying to inject to deployment

1/6/2022

I was testing out consul on my k3s cluster. After testing it out, I have removed the consul namespace and did a helm uninstall. Consul should be completely removed from the k3s cluster. However, I am getting the following error below for any application deployments. It seems like somewhere in the k3s cluster, there are some default consul settings to inject consul into the pods.

'Internal error occurred: failed calling webhook "consul-connect-injector.consul.hashicorp.com":
      Post "https://consul-connect-injector-svc.consul-system.svc:443/mutate?timeout=10s":
      service "consul-connect-injector-svc" not found'

When inspecting the replicaset manifest. no annotations were specified anywhere to do a consul inject. I've also tried to add in consul.hashicorp.com/connect-inject: "false" but I'm still getting similar results.

spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: testapp
      pod-template-hash: 7c8d846556
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/name:  testapp
        pod-template-hash: 7c8d846556
    spec:
      containers:
      - image: 
        imagePullPolicy: IfNotPresent
        name: testapp 
        ports:
        - containerPort: 8080
          protocol: TCP
        resources: {}
-- marvin lee jing rui
consul
kubernetes
kubernetes-helm

1 Answer

1/6/2022

I realized that after uninstalling the consul, the mutatingwebhooks of the consul are not removed. Went ahead to remove the 2 consul webhooks from kubectl get MutatingWebhookConfiguration <consul-webhook-name>. But I am not sure why the webhooks aren't removed when the uninstall command was ran.

-- marvin lee jing rui
Source: StackOverflow