ClusterIssuer with Cloudflare based DNS solver

2/8/2021

I'm trying to install the cert-manager ClusterIssuer on a AKS, and because the cluster is behind Azure Application Gateway I've gone down the route of using a DNS solver rather the HTTP. However, the challenge fails with an error calling the Cloudflare API. I've redacted emails and domains through the code snippets, the output of kubectl describe challenge rabt-cert-tls-g4mcl-1991965707-2468967546 is:

Events:
  Type     Reason        Age               From          Message
  ----     ------        ----              ----          -------
  Normal   Started       72s               cert-manager  Challenge scheduled for processing
  Warning  PresentError  3s (x5 over 71s)  cert-manager  Error presenting challenge: Cloudflare API Error for GET "/zones?name=<domain>"
            Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header

I have followed the guide at https://blog.darkedges.com/2020/05/04/cert-manager-kubernetes-cloudflare-dns-update/ and the issues at https://github.com/jetstack/cert-manager/issues/3021 and https://github.com/jetstack/cert-manager/issues/2384 but can't see any differences beyond the apiVersion of the issuer. I've checked this against the official documentation and there are no changes from what appears in these guides.

The relationship between ingress and cluster issuer seems fine; if I delete and recreate the ingress a new certificate, order and challenge are created. I've verified the secret is populated and I can print it to console, so it shouldn't be sending a blank string in the header. The token is valid, I can use the example CURL request from CloudFlare to check its validity.

Is there somewhere I can see logs and find out exactly what is being sent?

ClusterIssuer

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: ${CLOUDFLARE_API_TOKEN}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: rabt-letsencrypt
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <email>
    # ACME server URL for Let’s Encrypt’s staging environment.
    # The staging environment will not issue trusted certificates but is
    # used to ensure that the verification process is working properly
    # before moving to production
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: rabt-letsencrypt-key
    # Enable the HTTP-01 challenge provider
    # you prove ownership of a domain by ensuring that a particular
    # file is present at the domain
    solvers:
    - dns01:
         cloudflare:
           email: <email>
           apiTokenSecretRef:
             name: cloudflare-api-token-secret
             key: api-key

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rabt-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-protocol: https
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: rabt-letsencrypt
    cert-manager.io/acme-challenge-type: dns01
    appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
  tls:
  - hosts:
    - "*.rabt.<domain>"
    secretName: rabt-cert-tls
  rules:
  - host: "mq.rabt.<domain>"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: rabt-mq
            port:
              number: 15672
  - host: es.rabt.<domain>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: rabt-db-es-http
            port:
              number: 9200
  - host: "kibana.rabt.<domain>"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: rabt-kb-http
            port:
              number: 5601
-- Mark
cert-manager
cloudflare
kubernetes

1 Answer

2/11/2021

As Harsh Manvar guessed, it was an issue with the secret. I wasn't running the kubectl apply command through envsubst so it was encoding the literal string "${CLOUDFLARE_API_TOKEN}"

-- Mark
Source: StackOverflow