Edit This question used to be focused on the Kubernetes side of things. It is now apparent that the problem is at the Digitalocean Loadbalancer level.

I'm in the process of moving our service from Docker Swarm to a Kubernetes setup. The new K8S environment is up and running and I am starting to switch over traffic to the new K8S setup. However, when the traffic seems to be ramping up, it slows to a halt. The browser just spins for a while and then it loads snappily.

Running a simple curl -vvv https://thehostname.com and this happens

*   Trying 12.123.123.123...
* Connected to thehostname.com (12.123.123.123) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 594 certificates in /etc/ssl/certs
* ALPN, offering http/1.1

Then it pauses on that line for a while and after around 30 seconds it loads the rest of the request.

The symptom is that when the amount of traffic increases, the response time increases. It starts at 0.5 seconds, and then steadily increases to 30 seconds and is caped there. When I turn off traffic, the response time goes back to normal. The number of requests per second isn't more than 20-30 at most when this starts happening.

It seems that the act of opening a TCP connection is the slow part. I'm in contact with DigitalOcean support, but so far it has not yielded anything as it probably needs to be escalated.