I am trying to set up an Azure Kubernetes cluster with an HTTPS ingress controller for separate dev, staging, and prod environments. I have followed the Microsoft Azure guide on how to Create an HTTPS ingress controller on Azure Kubernetes Service (AKS) which allows me to set up an HTTPS ingress controller for a single namespace, but my end goal is to have separate namespaces for the dev, staging, and prod environments. According to the answers to this question, the way to do this is to have the ingress controller on one namespace (ingress in my case), and then separate ingress rules for each namespace (dev in my case).

Hence I setup the nginx ingress controller and the cert-manager on the ingress namespace:

# Add the ingress-nginx repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

# Use Helm to deploy an NGINX ingress controller
helm install nginx-ingress ingress-nginx/ingress-nginx \
    --namespace ingress \
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set controller.admissionWebhooks.patch.nodeSelector."beta\.kubernetes\.io/os"=linux

# Label the ingress-basic namespace to disable resource validation
kubectl label namespace ingress cert-manager.io/disable-validation=true

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install cert-manager jetstack/cert-manager \
  --namespace ingress \
  --version v0.16.1 \
  --set installCRDs=true \
  --set nodeSelector."kubernetes\.io/os"=linux \
  --set webhook.nodeSelector."kubernetes\.io/os"=linux \
  --set cainjector.nodeSelector."kubernetes\.io/os"=linux

I then create a cluster-issuer.yml file with the following:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: email@address.com
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx
          podTemplate:
            spec:
              nodeSelector:
                "kubernetes.io/os": linux

which I apply with

$ kubectl apply -f cluster-issuer.yml

Next I create an ingress rule on the dev namespace with the following ingress.yml file:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-dev
  namespace: dev
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    cert-manager.io/cluster-issuer: letsencrypt
    
spec:
  tls:
  - hosts:
    - domain.azure.com
    secretName: tls-secret-dev
  rules:
  - host: domain.azure.com
    http:
      paths:
      - backend:
          serviceName: my-service
          servicePort: 80
        path: /dev/my-service(/|$)(.*)

and apply it:

$ kubectl apply -f ingress.yml

Now I check to see whether a secret has been created:

$ kubectl get certificate -n dev
NAME             READY   SECRET           AGE
tls-secret-dev   False   tls-secret-dev   61s

So it seems that something went wrong when creating the secret. If I look at the certificate, it seems that a certificate is requested, but it never gets further than that:

$ kubectl describe certificate tls-secret -n dev
Name:         tls-secret-dev
Namespace:    dev
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1beta1
Kind:         Certificate
...
Status:
  Conditions:
    Last Transition Time:        2021-02-16T13:47:33Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2021-02-16T13:47:33Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  tls-secret-dev-6ngw8
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Issuing    70s   cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  70s   cert-manager  Stored new private key in temporary Secret resource "tls-secret-dev-6ngw8"
  Normal  Requested  70s   cert-manager  Created new CertificateRequest resource "tls-secret-dev-vtlbd"

Looking at the certificate request, an order is created:

$ kubectl describe certificaterequest tls-secret-dev-vtlbd -n dev
Name:         tls-secret-dev-vtlbd
Namespace:    dev
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: tls-secret-dev
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: tls-secret-dev-6ngw8
API Version:  cert-manager.io/v1beta1
Kind:         CertificateRequest
...
Status:
  Conditions:
    Last Transition Time:  2021-02-16T13:47:33Z
    Message:               Waiting on certificate issuance from order dev/tls-secret-dev-vtlbd-526778456: ""
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  OrderCreated  3m3s  cert-manager  Created Order resource dev/tls-secret-dev-vtlbd-526778456

Inspecting the order is where the trail seems to run cold:

$ kubectl describe order tls-secret-dev-vtlbd-526778456 -n dev
Name:         tls-secret-dev-vtlbd-526778456
Namespace:    dev
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: tls-secret-dev
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: tls-secret-dev-6ngw8
API Version:  acme.cert-manager.io/v1beta1
Kind:         Order
...
Status:
Events:  <none>

Question: How do I get the certificate manager to stop waiting on certificate issuance so I can finish setting up my HTTPS ingress controller?