I'm using standard procedure for enabling HTTPS termination for my application that is running on Kubernetes using: - Ingress nginx - AWS ELB classic - Cert Manager for Let's encrypt
I've used procedure described here: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes
I've been able to make Ingress work with HTTP before but I'm having problem with HTTPS where I'm getting following error when I try to cURL app URL:
$ curl -iv https://<SERVER>
* Rebuilt URL to: <SERVER>
* Trying <IP_ADDR>...
* Connected to <SERVER> (<IP_ADDR>) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: An unexpected TLS packet was received.
* Closing connection 0
curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.
This is what I currently have:
Cert manager running in kube-system namespace:
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
cert-manager-5f8db6f6c4-c4t4k 1/1 Running 0 2d1h
cert-manager-webhook-85dd96d87-rxc7p 1/1 Running 0 2d1h
cert-manager-webhook-ca-sync-pgq6b 0/1 Completed 2 2d1h
Ingress setup in ingress-nginx namespace:
$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/default-http-backend-587b7d64b5-ftws2 1/1 Running 0 2d1h
pod/nginx-ingress-controller-68bb4bfd98-zsz8d 1/1 Running 0 12h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/default-http-backend ClusterIP <IP_ADDR_1> <none> 80/TCP 2d1h
service/ingress-nginx NodePort <IP_ADDR_2> <none> 80:32327/TCP,443:30313/TCP 2d1h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/default-http-backend 1 1 1 1 2d1h
deployment.apps/nginx-ingress-controller 1 1 1 1 12h
Application and ingress in app namespace:
$ kubectl get all -n app
NAME READY STATUS RESTARTS AGE
pod/appserver-0 1/1 Running 0 2d1h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/appserver ClusterIP <IP_ADDR> <none> 22/TCP,80/TCP 2d1h
NAME DESIRED CURRENT AGE
statefulset.apps/appserver 1 1 2d1h
$ kubectl describe ingress -n app
Name: appserver
Namespace: app
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
letsencrypt-prod terminates <SERVER>
Rules:
Host Path Backends
---- ---- --------
<SERVER>
/ appserver:80 (<none>)
Annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
Events: <none>
This is how ingress resource looks like:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
name: appserver
namespace: app
spec:
tls:
- hosts:
- <SERVER>
secretName: letsencrypt-prod
rules:
- host: <SERVER>
http:
paths:
- backend:
serviceName: appserver
servicePort: 80
path: /
Additional checks that I've done:
Checked that certificates have been generated correctly:
$ kubectl describe cert -n app
Name: letsencrypt-prod
...
Status:
Conditions:
Last Transition Time: 2019-03-20T14:23:07Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Checked logs from cert-manager:
$ kubectl logs -f cert-manager-5f8db6f6c4-c4t4k -n kube-system
...
I0320 14:23:08.368872 1 sync.go:177] Certificate "letsencrypt-prod" for ingress "appserver" already exists
I0320 14:23:08.368889 1 sync.go:180] Certificate "letsencrypt-prod" for ingress "appserver" is up to date
I0320 14:23:08.368894 1 controller.go:179] ingress-shim controller: Finished processing work item "app/appserver"
I0320 14:23:12.548963 1 controller.go:183] orders controller: syncing item 'app/letsencrypt-prod-1237734172'
I0320 14:23:12.549608 1 controller.go:189] orders controller: Finished processing work item "app/letsencrypt-prod-1237734172"
Not really sure at this point what else might be worth of checking?
After all, it seems that problem was related with the fact on how I've done setup of listeners on ELB classic on AWS.
I've done following:
HTTP 80 -> HTTP <INGRESS_SVC_NODE_PORT_1>
HTTP 443 -> HTTP <INGRESS_SVC_NODE_PORT_2>
First mistake was that I've used HTTP instead of HTTPS for 443 port. When I tried to use HTTPS, I had to enter SSL certificates which didn't make sense to me since I'm doing SSL termination on Ingress level with Let's encrypt.
Therefore, following configuration of listener worked:
TCP 80 -> TCP <INGRESS_SVC_NODE_PORT_1>
TCP 443 -> TCP <INGRESS_SVC_NODE_PORT_2>
It appears that you are using legacy versions of the nginx-ingress with cert-manager, which could be the culprit to your error.
I suggest you try to deploy the nginx-ingress with Helm using the latest version (ensure that the image tag is v0.23.0 for the nginx-ingress-controller). Also make sure you are deploying your cert-manager with the latest version (--version v0.7.0) during your helm deployment. You should see three pods by default: cert-manager, cert-manager-cainjector, and cert-manager-webhook.
I had to disable the webhook because it was preventing the certificate from being issued despite providing it with the correct parameters, so you may have to do the same.
Hope this helps!