I run my application on Kubernetes 1.14.8.
All worked okay until yesterday (I have been using this for 3 months) but suddenly I get connection timeout in my application to database server located outside. Error message :
"HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@15be04e8 (This connection has been closed.)"
Also the pods seem to have problem communicating each other. Error message :
KubernetesClientException: Operation: [get] for kind: [Endpoints] with name: [membership-service] in namespace: [my-api] failed.
It seems like something within Kubernetes is not working properly. (maybe network related) However I can't seem to find where to look for debugging this. I skim through multiple Kubernetes logs and I see some
"read tcp 172.16.1.9:52410->10.96.0.1:443: read: connection timed out." messages "W0212 02:03:55.600088 6 reflector.go:302] pkg/mod/k8s.io/client-go@v0.0.0-20190612125919-78d2af7/tools/cache/reflector.go:98: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 172.16.1.133:58810->10.96.0.1:443: read: connection timed out") has prevented the request from succeeding"
but not sure how to fix this. (FYI, 10.96.0.1 is the IP address of default Kubernetes service)
Also I can't log-in to Kubernetes Dashboard. When I click 'SIGN IN' button, it fails with
"Get https://10.96.0.1:443/version: read tcp 172.16.1.9:52410->10.96.0.1:443: read: connection timed out" error.
Does anyone have similar experience?