We are trying to move our entire app infrastructure to Kubernetes and the last thing that is left to do is to configure TLS. We are using Kubernetes on DigitalOcean and our DNS is on Cloudflare.
For simplicity, we decided to go with a wild card certificate and followed these docs to accomplish it.
Here is what we have until now.
Issuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: cloudflare-ambassador-wcard
# namespace: ambassador
spec:
# ACME issuer configuration:
# `email` - the email address to be associated with the ACME account (make sure it's a valid one).
# `server` - the URL used to access the ACME server’s directory endpoint.
# `privateKeySecretRef` - Kubernetes Secret to store the automatically generated ACME account private key.
acme:
email: alex@priz.guru
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: cloudflare-ambassador-wcard-private
# List of challenge solvers that will be used to solve ACME challenges for the matching domains.
solvers:
- dns01:
cloudflare:
email: <my email - same as in Cloudflare>
apiKeySecretRef:
name: cloudflare-api-token-secret
key: api-token
selector:
dnsNames:
- 'app.priz.guru'
- 'appp.priz.guru'
- 'api.priz.guru'Certificate
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: priz-guru-cert
# Cert-Manager will put the resulting Secret in the same Kubernetes namespace as the Certificate.
namespace: ambassador
spec:
# Secret name to create, where the private key and certificate should be stored.
secretName: priz-guru-cert
# What Issuer to use for getting the certificate.
issuerRef:
name: cloudflare-ambassador-wcard
kind: ClusterIssuer
group: cert-manager.io
# Common name to be used on the Certificate.
commonName: "*.priz.guru"
# List of DNS subjectAltNames to be set on the Certificate.
dnsNames:
- 'appp.priz.guru'
- 'api.priz.guru'Cloudflare API token is configured in secrets.
When I try getting the certificate info, I see that it is not ready (for a very long time):
$ kubectl get certificate priz-guru-cert -n ambassador
NAME READY SECRET AGE
priz-guru-cert False priz-guru-cert 12hand looking at the logs, it seems like the certificate was approved, but it was not issued because there was another attempt within the last hour.
$ kubectl logs -l app=cert-manager,app.kubernetes.io/component=controller -n cert-manager
I0212 20:56:39.469298 1 trigger_controller.go:160] cert-manager/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="ambassador/priz-guru-cert" "retry_delay"=3599530749932
I0212 21:56:39.000546 1 trigger_controller.go:181] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="ambassador/priz-guru-cert" "message"="Issuing certificate as Secret does not exist" "reason"="DoesNotExist"
I0212 21:56:39.000597 1 conditions.go:190] Found status change for Certificate "priz-guru-cert" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2022-02-12 21:56:39.000591396 +0000 UTC m=+131965.533360318
I0212 21:56:39.569514 1 issuing_controller.go:265] cert-manager/certificates-issuing "msg"="Found a failed CertificateRequest from previous issuance, waiting for it to be deleted..." "key"="ambassador/priz-guru-cert" "resource_kind"="CertificateRequest" "resource_name"="priz-guru-cert-z2cnw" "resource_namespace"="ambassador" "resource_version"="v1"
I0212 21:56:39.578311 1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="ambassador/priz-guru-cert" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"priz-guru-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0212 21:56:39.601182 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "priz-guru-cert-9blj7" condition "Approved" to 2022-02-12 21:56:39.601171113 +0000 UTC m=+131966.133940054
I0212 21:56:39.618109 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "priz-guru-cert-9blj7" condition "Ready" to 2022-02-12 21:56:39.61810027 +0000 UTC m=+131966.150869178
I0212 21:56:39.635005 1 conditions.go:190] Found status change for Certificate "priz-guru-cert" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2022-02-12 21:56:39.63499362 +0000 UTC m=+131966.167762571
I0212 21:56:39.647445 1 trigger_controller.go:160] cert-manager/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="ambassador/priz-guru-cert" "retry_delay"=3599352583741
I0212 21:56:39.668068 1 trigger_controller.go:160] cert-manager/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="ambassador/priz-guru-cert" "retry_delay"=3599331976662Tried deleting everything and reapplying (after more than an hour). Same result.
How can I see what is the issue here? Are we even configuring it correctly?
UPDATE
Here is the description of the same certificate.
$ kubectl describe certificate priz-guru-cert -n ambassador
Name: priz-guru-cert
Namespace: ambassador
Labels: kustomize.toolkit.fluxcd.io/name=flux-system
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations: kustomize.toolkit.fluxcd.io/checksum: 8a875fb65b8d2a0d1ca76e552d21dca509e81ab7
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2022-02-12T09:56:21Z
Generation: 2
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:kustomize.toolkit.fluxcd.io/checksum:
f:labels:
.:
f:kustomize.toolkit.fluxcd.io/name:
f:kustomize.toolkit.fluxcd.io/namespace:
f:spec:
.:
f:commonName:
f:dnsNames:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:secretName:
Manager: kustomize-controller
Operation: Update
Time: 2022-02-12T09:56:21Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:lastFailureTime:
Manager: controller
Operation: Update
Time: 2022-02-12T22:56:39Z
Resource Version: 7912521
UID: deedc903-dc40-4e32-a4e8-91765cb33347
Spec:
Common Name: *.priz.guru
Dns Names:
appp.priz.guru
api.priz.guru
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: cloudflare-ambassador-wcard
Secret Name: priz-guru-cert
Status:
Conditions:
Last Transition Time: 2022-02-12T09:56:21Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 2
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2022-02-12T22:56:39Z
Message: The certificate request has failed to complete and will be retried: The CSR PEM requests a commonName that is not present in the list of dnsNames or ipAddresses. If a commonName is set, ACME requires that the value is also present in the list of dnsNames or ipAddresses: "*.priz.guru" does not exist in [appp.priz.guru api.priz.guru] or []
Observed Generation: 2
Reason: Failed
Status: False
Type: Issuing
Last Failure Time: 2022-02-12T22:56:39Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 46m (x15 over 13h) cert-manager Issuing certificate as Secret does not exist
Warning Failed 46m (x14 over 13h) cert-manager The certificate request has failed to complete and will be retried: The CSR PEM requests a commonName that is not present in the list of dnsNames or ipAddresses. If a commonName is set, ACME requires that the value is also present in the list of dnsNames or ipAddresses: "*.priz.guru" does not exist in [appp.priz.guru api.priz.guru] or []
Normal Generated 46m cert-manager Stored new private key in temporary Secret resource "priz-guru-cert-dmp62"
Normal Requested 46m cert-manager Created new CertificateRequest resource "priz-guru-cert-gmhfs"UPDATE - Trying to address "The CSR PEM requests a commonName that is not present in the list of dnsNames or ipAddresses"
Now the certificate config is:
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: priz-guru-cert
# Cert-Manager will put the resulting Secret in the same Kubernetes namespace as the Certificate.
namespace: ambassador
spec:
# Secret name to create, where the private key and certificate should be stored.
secretName: priz-guru-cert
# What Issuer to use for getting the certificate.
issuerRef:
name: cloudflare-ambassador-wcard
kind: ClusterIssuer
group: cert-manager.io
# Common name to be used on the Certificate.
# commonName: "*.priz.guru"
# List of DNS subjectAltNames to be set on the Certificate.
dnsNames:
- '*.priz.guru'Still not issuing with the following:
$ kubectl describe certificate priz-guru-cert -n ambassador
Name: priz-guru-cert
Namespace: ambassador
Labels: kustomize.toolkit.fluxcd.io/name=flux-system
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations: kustomize.toolkit.fluxcd.io/checksum: 5208aadd2a6d21e1d6f2f2dfc3f8d29a63990962
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2022-02-12T09:56:21Z
Generation: 7
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:kustomize.toolkit.fluxcd.io/checksum:
f:labels:
.:
f:kustomize.toolkit.fluxcd.io/name:
f:kustomize.toolkit.fluxcd.io/namespace:
f:spec:
.:
f:dnsNames:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:secretName:
Manager: kustomize-controller
Operation: Update
Time: 2022-02-12T23:58:13Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:lastFailureTime:
f:nextPrivateKeySecretName:
Manager: controller
Operation: Update
Time: 2022-02-12T23:59:15Z
Resource Version: 7933741
UID: deedc903-dc40-4e32-a4e8-91765cb33347
Spec:
Dns Names:
*.priz.guru
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: cloudflare-ambassador-wcard
Secret Name: priz-guru-cert
Status:
Conditions:
Last Transition Time: 2022-02-12T09:56:21Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 7
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2022-02-12T23:59:14Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 6
Reason: DoesNotExist
Status: True
Type: Issuing
Last Failure Time: 2022-02-12T23:58:14Z
Next Private Key Secret Name: priz-guru-cert-kkjxx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Generated 34m cert-manager Stored new private key in temporary Secret resource "priz-guru-cert-xx4kz"
Normal Requested 34m cert-manager Created new CertificateRequest resource "priz-guru-cert-lgcwt"
Warning Failed 34m cert-manager The certificate request has failed to complete and will be retried: Failed to wait for order resource "priz-guru-cert-lgcwt-3857888143" to become ready: order is in "errored" state: Failed to create Order: 400 urn:ietf:params:acme:error:malformed: Error creating new order :: Domain name "api.priz.guru" is redundant with a wildcard domain in the same request. Remove one or the other from the certificate request.
Normal Generated 30m cert-manager Stored new private key in temporary Secret resource "priz-guru-cert-q89c8"
Normal Requested 30m cert-manager Created new CertificateRequest resource "priz-guru-cert-wgqwf"
Normal Requested 22m cert-manager Created new CertificateRequest resource "priz-guru-cert-v4dj4"
Warning Failed 22m cert-manager The certificate request has failed to complete and will be retried: Failed to wait for order resource "priz-guru-cert-v4dj4-887490919" to become ready: order is in "errored" state: Failed to create Order: 400 urn:ietf:params:acme:error:malformed: Error creating new order :: Domain name "api.priz.guru" is redundant with a wildcard domain in the same request. Remove one or the other from the certificate request.
Normal Issuing 21m (x18 over 14h) cert-manager Issuing certificate as Secret does not exist
Normal Generated 21m cert-manager Stored new private key in temporary Secret resource "priz-guru-cert-kkjxx"
Normal Requested 21m cert-manager Created new CertificateRequest resource "priz-guru-cert-9w7pr"
Normal Requested 3m41s cert-manager Created new CertificateRequest resource "priz-guru-cert-2ntk9"