Setting up kubernetes with Let's Encrypt and Ingress - Gnutls_handshake() failed: An unexpected TLS packet was received

3/21/2019

I'm using standard procedure for enabling HTTPS termination for my application that is running on Kubernetes using: - Ingress nginx - AWS ELB classic - Cert Manager for Let's encrypt

I've used procedure described here: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes

I've been able to make Ingress work with HTTP before but I'm having problem with HTTPS where I'm getting following error when I try to cURL app URL:

$ curl -iv https://<SERVER>
* Rebuilt URL to: <SERVER>
*   Trying <IP_ADDR>...
* Connected to <SERVER> (<IP_ADDR>) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: An unexpected TLS packet was received.
* Closing connection 0
curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.

This is what I currently have:

Cert manager running in kube-system namespace:

$ kubectl get pods -n kube-system
    NAME                                     READY   STATUS      RESTARTS   AGE
    cert-manager-5f8db6f6c4-c4t4k            1/1     Running     0          2d1h
    cert-manager-webhook-85dd96d87-rxc7p     1/1     Running     0          2d1h
    cert-manager-webhook-ca-sync-pgq6b       0/1     Completed   2          2d1h

Ingress setup in ingress-nginx namespace:

$ kubectl get all -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
pod/default-http-backend-587b7d64b5-ftws2       1/1     Running   0          2d1h
pod/nginx-ingress-controller-68bb4bfd98-zsz8d   1/1     Running   0          12h

NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/default-http-backend   ClusterIP   <IP_ADDR_1>   <none>        80/TCP                       2d1h
service/ingress-nginx          NodePort    <IP_ADDR_2>   <none>        80:32327/TCP,443:30313/TCP   2d1h

NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/default-http-backend       1         1         1            1           2d1h
deployment.apps/nginx-ingress-controller   1         1         1            1           12h

Application and ingress in app namespace:

$ kubectl get all -n app
NAME              READY   STATUS    RESTARTS   AGE
pod/appserver-0   1/1     Running   0          2d1h

NAME                TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
service/appserver   ClusterIP   <IP_ADDR>   <none>        22/TCP,80/TCP   2d1h

NAME                         DESIRED   CURRENT   AGE
statefulset.apps/appserver   1         1         2d1h

$ kubectl describe ingress -n app
Name:             appserver
Namespace:        app
Address:
Default backend:  default-http-backend:80 (<none>)
TLS:
  letsencrypt-prod terminates <SERVER>
Rules:
  Host                      Path  Backends
  ----                      ----  --------
  <SERVER>
                            /   appserver:80 (<none>)
Annotations:
  certmanager.k8s.io/cluster-issuer:  letsencrypt-prod
  kubernetes.io/ingress.class:        nginx
Events:                               <none>

This is how ingress resource looks like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
  name: appserver
  namespace: app
spec:
  tls:
  - hosts:
    - <SERVER>
    secretName: letsencrypt-prod
  rules:
  - host: <SERVER>
    http:
      paths:
      - backend:
          serviceName: appserver
          servicePort: 80
        path: /

Additional checks that I've done:

Checked that certificates have been generated correctly:

$ kubectl describe cert -n app
Name:         letsencrypt-prod
...
Status:
  Conditions:
    Last Transition Time:  2019-03-20T14:23:07Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready

Checked logs from cert-manager:

$ kubectl logs -f cert-manager-5f8db6f6c4-c4t4k -n kube-system
...
I0320 14:23:08.368872       1 sync.go:177] Certificate "letsencrypt-prod" for ingress "appserver" already exists
I0320 14:23:08.368889       1 sync.go:180] Certificate "letsencrypt-prod" for ingress "appserver" is up to date
I0320 14:23:08.368894       1 controller.go:179] ingress-shim controller: Finished processing work item "app/appserver"
I0320 14:23:12.548963       1 controller.go:183] orders controller: syncing item 'app/letsencrypt-prod-1237734172'
I0320 14:23:12.549608       1 controller.go:189] orders controller: Finished processing work item "app/letsencrypt-prod-1237734172"

Not really sure at this point what else might be worth of checking?

-- Bakir Jusufbegovic
cert-manager
kubernetes
nginx-ingress

2 Answers

3/22/2019

After all, it seems that problem was related with the fact on how I've done setup of listeners on ELB classic on AWS.

I've done following:

HTTP 80 -> HTTP <INGRESS_SVC_NODE_PORT_1>
HTTP 443 -> HTTP <INGRESS_SVC_NODE_PORT_2>

First mistake was that I've used HTTP instead of HTTPS for 443 port. When I tried to use HTTPS, I had to enter SSL certificates which didn't make sense to me since I'm doing SSL termination on Ingress level with Let's encrypt.

Therefore, following configuration of listener worked:

TCP 80 -> TCP <INGRESS_SVC_NODE_PORT_1>
TCP 443 -> TCP <INGRESS_SVC_NODE_PORT_2>
-- Bakir Jusufbegovic
Source: StackOverflow

3/21/2019

It appears that you are using legacy versions of the nginx-ingress with cert-manager, which could be the culprit to your error.

I suggest you try to deploy the nginx-ingress with Helm using the latest version (ensure that the image tag is v0.23.0 for the nginx-ingress-controller). Also make sure you are deploying your cert-manager with the latest version (--version v0.7.0) during your helm deployment. You should see three pods by default: cert-manager, cert-manager-cainjector, and cert-manager-webhook.

I had to disable the webhook because it was preventing the certificate from being issued despite providing it with the correct parameters, so you may have to do the same.

Hope this helps!

-- Frank Yucheng Gu
Source: StackOverflow