How do we debug networking issues within istio pods?

12/26/2019

I am working on setting up istio in my kubernetes cluster.

I downloaded istio-1.4.2 and installed demo profile and did manual sidecar injection.

But when I check sidecar pod logs, I am getting the below error.

2019-12-26T08:54:17.694727Z     error   k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:98: Failed to list *v1beta1.MutatingWebhookConfiguration: Get https://10.96.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations?fieldSelector=metadata.name%3Distio-sidecar-injector&limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused

It seems to be the networking issue, but could you please let me know what it is trying to do exactly?

Is there a way to get more logs than just 'connection refused'?

How do we verify networking issues between istio pods. It seems I cannot run 'wget', 'curl', 'tcpdump', 'netstat' etc within istio sidecar pod to debug further.

All the pods in kube-system namespace are working fine.

-- Kalyan Kumar
istio
kubernetes

1 Answer

12/26/2019

Check what port your API Server is serving https traffic(controlled by this flag --secure-port int Default: 6443). It may be 6443 instead of 443. Check what is the value of server in your kubeconfig and are you able to connect to your kubernetes via kubectl using that kubeconfig.

Another thing to check is whether you have network policy attached to the namespace which blocks egress traffic.

And you could use an ephemeral container to debug issue with the sidecar

https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/

https://github.com/aylei/kubectl-debug

-- Arghya Sadhu
Source: StackOverflow