I'm currently running Sentry in Kubernetes with auto certificate generation using let's encrypt and cert-manager. When Sentry attempts to send an error to the sentry server, the following error is thrown:
urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)> (url: https://example.host.com/)I have verified that the correct python packages for 2.7.15 have been installed. Packages include certifi, urllib2 along with the dependencies.
Turning off TLS Verification works, but this is a last resort. Security is very important even though this is an internally hosted service.
It has been my experience that even the most up-to-date ca-certificates packages sometimes don't contain all 3 Let's Encrypt certificates. The solution(?) is to download them into the "user-controlled" certificate directory (often /usr/local/share/ca-certificates) and then re-run update-ca-certificates:
# the first one very likely is already in your chain,
# but including it here won't hurt anything
for i in isrgrootx1.pem.txt lets-encrypt-x3-cross-signed.pem.txt letsencryptauthorityx3.pem.txt
do
curl -vko /usr/local/share/ca-certificates/`basename $i .pem.txt`.crt \
https://letsencrypt.org/certs/$i
done
update-ca-certificates
The ideal outcome would be to do that process for every Node in your cluster, and then volume mount the actual ssl directory into the containers, so every container benefits from the latest certificates. However, I would guess just doing it in the affected containers could work, too.