Kubernetes 1.14.8 - Random connection timed out

2/12/2020

I run my application on Kubernetes 1.14.8.

All worked okay until yesterday (I have been using this for 3 months) but suddenly I get connection timeout in my application to database server located outside. Error message :

"HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@15be04e8 (This connection has been closed.)"

Also the pods seem to have problem communicating each other. Error message :

KubernetesClientException: Operation: [get] for kind: [Endpoints] with name: [membership-service] in namespace: [my-api] failed.

It seems like something within Kubernetes is not working properly. (maybe network related) However I can't seem to find where to look for debugging this. I skim through multiple Kubernetes logs and I see some

"read tcp 172.16.1.9:52410->10.96.0.1:443: read: connection timed out." messages "W0212 02:03:55.600088 6 reflector.go:302] pkg/mod/k8s.io/client-go@v0.0.0-20190612125919-78d2af7/tools/cache/reflector.go:98: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 172.16.1.133:58810->10.96.0.1:443: read: connection timed out") has prevented the request from succeeding"

but not sure how to fix this. (FYI, 10.96.0.1 is the IP address of default Kubernetes service)

Also I can't log-in to Kubernetes Dashboard. When I click 'SIGN IN' button, it fails with

"Get https://10.96.0.1:443/version: read tcp 172.16.1.9:52410->10.96.0.1:443: read: connection timed out" error.

Does anyone have similar experience?

-- hinewwiner
kubernetes

0 Answers