I have a cluster that was recently upgraded. Since then none of the pods running in the cluster are able to get a response from https://kubernetes.default/healthz.

To be clear, they can resolve the URL, but it constantly times out of comes back with connection refused.

I have verified that the api-server is running as I can get a response from http://localhost:8080/healthz, but I can't get anything from within a pod.

I've checked all the scripts and configs and compared them to the other clusters that were upgraded at the same time and there appears to be nothing different.

I'm sure it's something small I've overlooked but I don't know where else to look.

Additional information:

• setup was with kops
• runs in AWS (not the managed services but raw)
• the above queries work as expected in other clusters
• upgrade was from 1.11 to 1.13 (yes, I know that is frowned upon. At least now I do.)
• other clusters had same upgrade path

[edit]

providing /etc/resolv.conf

cat /etc/resolv.conf
nameserver 100.64.0.10
search jenkins.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal
options ndots:5