I am following this tutorial to use a trusted TLS certificate in my ingress rules. All works, however
kubectl describe certificate quickstart-example-tlsgives & gets stuck at Created new CertificateRequest resource
....
Status:
Conditions:
Last Transition Time: 2022-02-08T10:00:01Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2022-02-08T10:00:02Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: True
Type: Issuing
Next Private Key Secret Name: quickstart-example-tls-d87r6
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 21m cert-manager Issuing certificate as Secret does not exist
Normal Generated 21m cert-manager Stored new private key in temporary Secret resource "quickstart-example-tls-d87r6"
Normal Requested 21m cert-manager Created new CertificateRequest resource "quickstart-example-tls-kxwqd"First let me share my ingress rules and Issuers:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: flask-ingress
spec:
ingressClassName: nginx
rules:
- host: www.<Domain>.net
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: flask
port:
number: 5000I set up DNS (in digital ocean) rules so curling (via web) works.
curl "http://www.<Domain>.net"
<p>Hello, World!</p>% Here my issuer file:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-staging
namespace: default
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: <MY-EMAIL>
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- http01:
ingress:
class: nginx# kubectl get issuer
NAME READY AGE
letsencrypt-staging True 26sNow I update my ingress rule and start the challenge:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: flask-ingress
annotations:
cert-manager.io/issuer: letsencrypt-staging
spec:
tls:
- hosts:
- www.<DOMAIN>.net
secretName: quickstart-example-tls
ingressClassName: nginx
rules:
- host: www.<DOMAIN>.net
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: flask
port:
number: 5000#kubectl get certificate
NAME READY SECRET AGE
quickstart-example-tls False quickstart-example-tls 88sHere the detailed output:
#kubectl describe certificate quickstart-example-tls
Name: quickstart-example-tls
Namespace: default
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2022-02-08T10:34:40Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:ownerReferences:
.:
k:{"uid":"<UID>"}:
.:
f:apiVersion:
f:blockOwnerDeletion:
f:controller:
f:kind:
f:name:
f:uid:
f:spec:
.:
f:dnsNames:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:secretName:
f:usages:
f:status:
.:
f:conditions:
f:nextPrivateKeySecretName:
Manager: controller
Operation: Update
Time: 2022-02-08T10:34:41Z
Owner References:
API Version: networking.k8s.io/v1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: flask-ingress
UID: <UID>
Resource Version: 321990
UID: <UID>
Spec:
Dns Names:
www.<DOMAIN>.net
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: letsencrypt-staging
Secret Name: quickstart-example-tls
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2022-02-08T10:34:40Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: True
Type: Issuing
Last Transition Time: 2022-02-08T10:34:40Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: False
Type: Ready
Next Private Key Secret Name: quickstart-example-tls-w4psf
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 2m14s cert-manager Issuing certificate as Secret does not exist
Normal Generated 2m13s cert-manager Stored new private key in temporary Secret resource "quickstart-example-tls-w4psf"
Normal Requested 2m13s cert-manager Created new CertificateRequest resource "quickstart-example-tls-znhlv"So this is were things get stucked and IDK why.
# kubectl describe certificaterequest quickstart-example-tls
Name: quickstart-example-tls-znhlv
Namespace: default
Labels: <none>
Annotations: cert-manager.io/certificate-name: quickstart-example-tls
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: quickstart-example-tls-w4psf
API Version: cert-manager.io/v1
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2022-02-08T10:34:41Z
Generate Name: quickstart-example-tls-
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:cert-manager.io/certificate-name:
f:cert-manager.io/certificate-revision:
f:cert-manager.io/private-key-secret-name:
f:generateName:
f:ownerReferences:
.:
k:{"uid":"<UID>"}:
.:
f:apiVersion:
f:blockOwnerDeletion:
f:controller:
f:kind:
f:name:
f:uid:
f:spec:
.:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:request:
f:usages:
f:status:
.:
f:conditions:
Manager: controller
Operation: Update
Time: 2022-02-08T10:34:41Z
Owner References:
API Version: cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: quickstart-example-tls
UID: <UID>
Resource Version: 322003
UID: <UID>
Spec:
Extra:
authentication.kubernetes.io/pod-name:
cert-manager-6d8d028374dbb-f43n7
authentication.kubernetes.io/pod-uid:
<UID>
Groups:
system:serviceaccounts
system:serviceaccounts:cert-manager
system:authenticated
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: letsencrypt-staging
Request: <Request ID>
UID: <UID>
Usages:
digital signature
key encipherment
Username: system:serviceaccount:cert-manager:cert-manager
Status:
Conditions:
Last Transition Time: 2022-02-08T10:34:41Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2022-02-08T10:34:43Z
Message: Waiting on certificate issuance from order default/quickstart-example-tls-znhlv-138543614: "pending"
Reason: Pending
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal cert-manager.io 4m54s cert-manager Certificate request has been approved by cert-manager.io
Normal OrderCreated 4m53s cert-manager Created Order resource default/quickstart-example-tls-znhlv-138543614Having a look at the challenge, all seems to work.
# kubectl describe challenges --all-namespaces
Name: quickstart-example-tls-znhlv-138543614-1891152753
Namespace: default
Labels: <none>
Annotations: <none>
API Version: acme.cert-manager.io/v1
Kind: Challenge
Metadata:
Creation Timestamp: 2022-02-08T10:34:44Z
Finalizers:
finalizer.acme.cert-manager.io
Generation: 1
Managed Fields:
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizer.acme.cert-manager.io":
f:ownerReferences:
.:
k:{"uid":"<UID>"}:
.:
f:apiVersion:
f:blockOwnerDeletion:
f:controller:
f:kind:
f:name:
f:uid:
f:spec:
.:
f:authorizationURL:
f:dnsName:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:key:
f:solver:
.:
f:http01:
.:
f:ingress:
.:
f:class:
f:token:
f:type:
f:url:
f:wildcard:
f:status:
.:
f:presented:
f:processing:
f:reason:
f:state:
Manager: controller
Operation: Update
Time: 2022-02-08T10:34:46Z
Owner References:
API Version: acme.cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: Order
Name: quickstart-example-tls-znhlv-138543614
UID: <UID>
Resource Version: 322038
UID: <UID>
Spec:
Authorization URL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/<NR>
Dns Name: www.<DOMAIN>.net
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: letsencrypt-staging
Key: <SOMEKEY>
Solver:
http01:
Ingress:
Class: nginx
Token: <SOMEToken>
Type: HTTP-01
URL: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/<NR>/<CODE>
Wildcard: false
Status:
Presented: true
Processing: true
Reason: Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://www.<DOMAIN>.net/.well-known/acme-challenge/<SOMEToken>': Get "http://www.<DOMAIN>.net/.well-known/acme-challenge/<SOMEToken>": EOF
State: pending
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 6m18s cert-manager Challenge scheduled for processing
Normal Presented 6m17s cert-manager Presented challenge using HTTP-01 challenge mechanism#curl "http://www.<DOMAIN>.net/.well-known/acme-challenge/<SOMEToken>"
<SOMEKEY>% All seems to be fine, but I do not get the certificate?
A couple of additional things:
#kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-6d8d6b5dbb-f4xn7 1/1 Running 0 47h
cert-manager-cainjector-d6cbc4d9-g6j8c 1/1 Running 0 47h
cert-manager-webhook-85fb68c79b-sv8pb 1/1 Running 0 47hLooking add the logs from cert-manager, I get that get request failed although it works actually.
# kubectl logs cert-manager-6d8d6b5dbb-f4xn7 -n cert-manager
E0208 15:07:31.660877 1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://www.<DOMAIN>.net/.well-known/acme-challenge/<SOMEToken>': Get \"http://www.<DOMAIN>.net/.well-known/acme-challenge/<SOMEToken>\": EOF" "dnsName"="www.<DOMAIN>.net" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-znhlv-138543614-1891152753" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" @Rychu linked me to the answer here. This goes also back to this post step 5
For my case I did
Added a DNS entry on digital ocean for "workaround.<"DOMAIN">.net" to point to my load balancer and then I updated my ingress_controller:
cat ingress_nginx_svc.yamlapiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: 'true'
service.beta.kubernetes.io/do-loadbalancer-hostname: "workaround.<DOMAIN>.net"
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/version: 1.0.4
helm.sh/chart: ingress-nginx-4.0.6
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
externalTrafficPolicy: Cluster
ports:
- appProtocol: http
name: http
nodePort: 31799
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
nodePort: 32533
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: LoadBalancerkubectl apply -f ingress_nginx_svc.yamlThen you should see something like this
# kubectl describe certificate
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 15m cert-manager Issuing certificate as Secret does not exist
Normal Generated 15m cert-manager Stored new private key in temporary Secret resource "quickstart-example-tls-zj57f"
Normal Requested 15m cert-manager Created new CertificateRequest resource "quickstart-example-tls-gc599"
Normal Issuing 8m3s cert-manager Issuing certificate as Secret was previously issued by Issuer.cert-manager.io/letsencrypt-staging
Normal Reused 8m2s cert-manager Reusing private key stored in existing Secret resource "quickstart-example-tls"
Normal Requested 8m2s cert-manager Created new CertificateRequest resource "quickstart-example-tls-55xcw"
Normal Issuing 7m33s (x2 over 11m) cert-manager The certificate has been successfully issued
That solved my problem! Thanks again @Rychu for pointing me to that