I have a service running on kubernetes behind a service using cluster ip exposing the port 443 and an nginx ingress (using the default self signed cert) which exposes this service.

The service works completely fine, I can do everything I normally do with the application but I get this issue every now and then when using python requests where I can't make requests due to SSL issues:

Traceback (most recent call last):
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/urllib3/connection.py", line 421, in connect
    tls_in_tls=tls_in_tls,
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 429, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='host', port=443): Max retries exceeded with url: /v4/workflows/query-document/test/runs/ (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/barmstrong/repos/amp/scripts/post_runs.py", line 14, in <module>
    response = requests.post(WORKFLOW_URL, files=SOURCE_DATA, headers=HEADERS, verify=False, timeout=30, data={'tags': ['test1']})
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/barmstrong/.virtualenvs/amp/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='host', port=443): Max retries exceeded with url: /v4/workflows/query-document/test/runs/ (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:852)'),))

This is unusual because literally a day ago it worked fine, if I uninstall the ingress controller and reinstall it works as expected again. Also one or 2 of my post requests will go through successfully before encountering this error. I have verify SSL requests disabled in my requests post request.

response = requests.post(WORKFLOW_URL, files=SOURCE_DATA, headers=HEADERS, verify=False, timeout=30, data={'tags': ['test1']})

The two successful requests logs from the ingress controller pod followed by the failed one (occassionaly one or two will go through before I see this error):

10.100.1.1 - - [07/May/2021:10:24:19 +0000] "POST /v4/workflows/query-document/test/runs/ HTTP/1.1" 201 118 "-" "python-requests/2.25.1" 703 1.083 [amp-amp-server-443] [] 10.100.1.81:8884 118 1.083 201 8988570ca7d94698de30ae96dd0f0e85
10.0.0.39 - - [07/May/2021:10:24:20 +0000] "POST /v4/workflows/query-document/test/runs/ HTTP/1.1" 201 118 "-" "python-requests/2.25.1" 703 0.960 [amp-amp-server-443] [] 10.100.1.81:8884 118 0.961 201 9b66042a892cbb6d2c66563879734fef
10.0.0.39 - - [07/May/2021:10:24:20 +0000] "\x16\x03\x01\x02\x00\x01\x00\x01\xFC\x03\x031U\x81\xD1_ `\xB9\x13\xFD\xF4\xBEX\x03\xC1\x8D\xD7\xA2\x9A\xB0`" 400 150 "-" "-" 0 0.132 [] [] - - - - 3489cf9059d14dcb7be5051aa5d19fbc

Does anyone have any idea what is going on here and why it initially works but some days later requests fail?