Looking for debugging advice on SSL errors from EKS using varnish

1/17/2020

I know this is somewhat specific of a question, but I'm having a problem I can't seem to track down. I have a single pod deployed to EKS - the pod contains a python app, and a varnish reverse caching proxy. I'm serving chunked json (that is, streaming lines of json, a la http://jsonlines.org/), and it can be multiple GB of data.

The first time I make a request, and it hits the python server, everything acts correctly. It takes (much) longer than the cached version, but the entire set of json lines is downloaded. However, now that it's cached in varnish, if I use curl, I get:

curl: (56) GnuTLS recv error (-110): The TLS connection was non-properly terminated.

or

curl: (56) GnuTLS recv error (-9): A TLS packet with unexpected length was received.

The SSL is terminated at the ELB, and when I use curl from the proxy container itself (using curl http://localhost?....), there is no problem.

The hard part of this is that the problem is somewhat intermittent.

If there is any advice in terms of clever varnishlog usage, or anything of the same ilk on AWS, I'd be much obliged.

Thanks!

-- Hoopes
amazon-eks
aws-eks
kubernetes
varnish

1 Answer

2/18/2020

Because TLS is terminated on your ELB loadbalancers, the connection between should be in plain HTTP.

The error is probably not coming from Varnish, because Varnish currently doesn't handle TLS natively. I'm not sure if varnishlog can give you better insights in what is actually happening.

Checklist

The only checklist I can give you is the following:

  • Make sure the certificate you're using is valid
  • Make sure you're connecting to your target group over HTTP, not HTTPS
  • If you enable the PROXY protocol on your ELB, make sure Varnish has a -a listener that listens for PROXY protocol requests, on top of regular HTTP requests.

Debugging

Perform top-down debugging:

  • Increase the verbosity of your cURL calls and try to get more information about the error
  • Try accessing the logs of your ELB and get more details there
  • Get more information from your EKS logs
  • And finally, perform a varnislog -g request -q "ReqUrl eq '/your-url'" to get a full Varnishlog for a specific URL
-- Thijs Feryn
Source: StackOverflow