Why is my k8s Nginx ingress controller serving two certificates (one of which is a Kubernetes Fake Certificate)?

2/4/2021

We are running a AKS Kubernetes cluster on Azure. I'm using the "NGINX Ingress Controller" and "cert-manager" for routing and certificate generation (through Let's Encrypt). I followed the basic setup advice from the Microsoft documentation: https://docs.microsoft.com/en-us/azure/aks/ingress-tls

When visiting our page in a web-browser, we notice nothing out of the ordinary at first - HTTPS was working fine. The browser can validate the Let's Encrypt certificate. However, we noticed later on that the ingress controller actually serves two certificates (one of which has a common name: "Kubernetes Ingress Controller Fake Certificate" and an Alternative name: "ingress.local"): https://www.ssllabs.com/ssltest/analyze.html?d=test-aks-ingress.switzerlandnorth.cloudapp.azure.com&hideResults=on

Long story short - yesterday, I tried everything from re-installing the Nginx-ingress and cert-manager to starting a new Azure Kubernetes Service from scratch, but every time I end up in the same situation.

I have read many of the discussions from people experiencing similar problems. Typically, they are a bit different though, as they don't actually see a valid certificate at all. I confirmed that we are using the production Let's Encrypt ClusterIssuer:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ***@***
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx
          podTemplate:
            spec:
              nodeSelector:
                "kubernetes.io/os": linux

I also created a new test-app with test-ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - test-aks-ingress.switzerlandnorth.cloudapp.azure.com
    secretName: tls-secret
  rules:
  - host: test-aks-ingress.switzerlandnorth.cloudapp.azure.com
    http:
      paths:
      - backend:
          serviceName: aks-helloworld-one
          servicePort: 80
        path: /hello-world-one(/|$)(.*)

From my understanding there is usually some issue with the secret for the people that have reported this previously. Here I assume that the ClusterIssuer will generate the relevant certificates and store them in tls-secret, which has been generated automatically:

Name:         tls-secret                                                                       
Namespace:    test                                                                             
Labels:       <none>                                                                           
Annotations:  cert-manager.io/alt-names: test-aks-ingress.switzerlandnorth.cloudapp.azure.com  
              cert-manager.io/certificate-name: tls-secret                                     
              cert-manager.io/common-name: test-aks-ingress.switzerlandnorth.cloudapp.azure.com
              cert-manager.io/ip-sans:                                                         
              cert-manager.io/issuer-group: cert-manager.io                                    
              cert-manager.io/issuer-kind: ClusterIssuer                                       
              cert-manager.io/issuer-name: letsencrypt                                         
              cert-manager.io/uri-sans:                                                        
                                                                                               
Type:  kubernetes.io/tls                                                                       
                                                                                               
Data                                                                                           
====                                                                                           
tls.crt:  3530 bytes                                                                           
tls.key:  1675 bytes  

Maybe what I am still confused about is the different secrets / certificates at play here. The cert-manager operates in the cert-manager namespace and creates a letsencrypt secret there, while my test-setup is running everything else in a test namespace (including the ingress controller).

UPDATE But what is the actual problem here? Everything "just works" in a normal browser, right? Unfortunately, the real problem is that connections do not work for a specific client application, which may not have SNI support.

Is there a way to not have a default certificate? How would I change the configuration here to provide the "Let's Encrypt" signed certificate by default - is that possible?

-- Chris
azure
cert-manager
kubernetes
nginx-ingress
ssl

1 Answer

2/4/2021

It is expected behavior. By default Ingress controller creates self-signed certificate with CN indicating it's fake one. This is used when a request doesn't match to rules defined in Ingress. So when we access this URL from browser, it returns correct certificate but with openssl s_client without servername field, it doesn't match the rule defined in Ingress and goes to default backend and returns self-signed certificate.

You can also specify default certificate for Ingress. Refer https://github.com/kubernetes/ingress-nginx/issues/4674 for more details.

-- Rushikesh
Source: StackOverflow