TLS certification expired only for some users

9/30/2021

I have a k8s cluster with an ingress nginx as a reverse proxy. I am using letsencrypt to generate TLS certificate

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ******
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx

Everything worked fine for months. Today,

$ curl -v --verbose https://myurl

returns

* Rebuilt URL to: https://myurl/
*   Trying 51.103.58.**...
* TCP_NODELAY set
* Connected to myurl (51.103.58.**) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

For 2 other people on my team, error is the same and I have the same error when I use Postman (expired certificate).

But for another one, we get no error :

*   Trying 51.103.58.**...
* TCP_NODELAY set
* Connected to myurl (51.103.58.**) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=myurl
*  start date: Jul 24 07:15:13 2021 GMT
*  expire date: Oct 22 07:15:11 2021 GMT
*  subjectAltName: host "myurl" matched cert's "myurl"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd9be00d600)
> GET / HTTP/2
> Host: myurl
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx/1.19.1
< date: Thu, 30 Sep 2021 16:11:23 GMT
< content-type: application/json; charset=utf-8
< content-length: 56
< vary: Origin, Accept-Encoding
< access-control-allow-credentials: true
< x-xss-protection: 1; mode=block
< x-frame-options: DENY
< strict-transport-security: max-age=15724800; includeSubDomains
< x-download-options: noopen
< x-content-type-options: nosniff
< etag: W/"38-3eQD3G7Y0vTkrLR+ExD2u5BSsMc"
<
* Connection #0 to host myurl left intact
{"started":"2021-09-30T13:30:30.912Z","uptime":9653.048}* Closing connection 0

When I use my web browser to go to the website, everything works fine and the certificate is presented as valid and for now, I get no error in prod or staging environment. (same error on staging)

Has anyone an explanation on this ?

-- soling
kubernetes
ssl

3 Answers

9/30/2021

Even if not K8S related, main explanation is contained in : https://stackoverflow.com/questions/69397218/sudden-openssl-error-messages-error14090086-using-file-get-contents . I complete it with K8S related here.

I fixed the same issue by upgarding my certbot and reissuing certificate with --preferred-chain 'ISRG Root X1'

you can do the same with options in the yaml of the cert issuer : see here : https://cert-manager.io/docs/configuration/acme/#use-an-alternative-certificate-chain

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    preferredChain: "ISRG Root X1"
-- MattVon
Source: StackOverflow

9/30/2021

Warning! Please plan OS upgrade path. The below advice should be applied only in emergency situation to quickly fix a critical system.

Your team missed OS update or ca-certificates package update. Below solution works on old Debian/Ubuntu systems.

First check if you have offending DST Root CA X3 cert present:

# grep X3 /etc/ca-certificates.conf 
mozilla/DST_Root_CA_X3.crt

Make sure the client OS have the proper ISRG Root X1 present too:

# grep X1 /etc/ca-certificates.conf 
mozilla/ISRG_Root_X1.crt

This is going to disable X3:

# sed -i '/^mozilla\/DST_Root_CA_X3/s/^/!/' /etc/ca-certificates.conf && update-ca-certificates -f

Try curl https://yourdomain now, should pass.

Again, plan an upgrade please.

-- gertas
Source: StackOverflow

10/1/2021

This is related to the expired DST Root CA X3, which expired Sep 30 14:01:15 2021 GMT.

The DST CA Root X3 certificate is part of the "cacert-bundle". As of today the "cacert-bundle" can be found here: https://curl.se/docs/caextract.html as part of the bundle https://curl.se/ca/cacert.pem.

The expired certificate is:

Certificate:
    Data:
    Version: 3 (0x2)
    Serial Number:
    44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
    Signature Algorithm: sha1WithRSAEncryption
    Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Validity
    Not Before: Sep 30 21:12:19 2000 GMT
    Not After : Sep 30 14:01:15 2021 GMT
    Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
    Subject Public Key Info:
    Public Key Algorithm: rsaEncryption
    Public-Key: (2048 bit)

Which is used to verify peer in curl calls to websites using Let's Encrypt issued certificates.

Here's a detailed solution to your problem: https://stackoverflow.com/a/69411107/1549092

Let's Encrypt formal address of the issue can be found here: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

-- GTodorov
Source: StackOverflow