Out of nowhere one of our API servers has started to fail with the following error:

http: TLS handshake error from 172.23.88.213:17244: EOF

It throws this error for every single node in the cluster, thus failing to start. This started happening this morning with no changes to any infrastructure.

Things I've tried that haven't helped:

• Manually restart the weave docker container on the master node.
• Manually kill and reschedule the api-server.
• Manually restart the Docker daemon.
• Manually restarted the kubelet service.
• Check all SSL certs are valid, which they are.
• Check inodes, thousands free.
• Ping IP addresses of other nodes in cluster, all return ok with 0 packet loss.
• Check journalctl and systemctl logs of kubelet services and the only significant errors I see are related to TLS handshake error.

Cluster specs:

Cloud provider: AWS
Kubernetes version: 1.11.6
Kubelet version: 1.11.6
Kops version: 1.11

I'm at a bit of a loss as to how to debug this further.