I followed this tutorial to get let's encrypt in kubernetes : https://github.com/ahmetb/gke-letsencrypt/blob/master/
I encountered some problems, cert-manager don't create the needed secret. Could you help me please to resolve this problem ?
Cert-manager ERRORS :
Found status change for Certificate "mydomain.fr" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-11-06 17:37:20.683089649 +0000 UTC m=+5887.364224968
Error preparing issuer for certificate coffeer-ci/mydomain.fr: http-01 self check failed for domain "mydomain.fr"
[coffeer-ci/mydomain.fr] Error getting certificate 'domain-tls': secret "domain-tls" not found
Here is my kubernetes objects :
kubectl -n kube-system describe pod cert-manager
Name: cert-manager-7bb46cc6b-scqrp
Namespace: kube-system
Node: gke-inkubator-default-pool-68c0309d-b86b/10.132.0.3
Start Time: Tue, 06 Nov 2018 16:59:10 +0100
Labels: app=cert-manager
pod-template-hash=366027726
release=cert-manager
Annotations: <none>
Status: Running
IP: 10.16.1.132
Controlled By: ReplicaSet/cert-manager-7bb46cc6b
Containers:
cert-manager:
Container ID: docker://d4795cfa85aacd2cbd0c5fd51246c436e3cf953632f4ca4a26e683c5867bf113
Image: quay.io/jetstack/cert-manager-controller:v0.5.0
Image ID: docker-pullable://quay.io/jetstack/cert-manager-controller@sha256:fd89c3c33fd89ffe0a9f91df2f54423397058d4180eccfe90b831859ba46b6e5
Port: <none>
Host Port: <none>
Args:
--cluster-resource-namespace=$(POD_NAMESPACE)
--leader-election-namespace=$(POD_NAMESPACE)
State: Running
Started: Tue, 06 Nov 2018 16:59:13 +0100
Ready: True
Restart Count: 0
Environment:
POD_NAMESPACE: kube-system (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from cert-manager-token-9ck7b (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
cert-manager-token-9ck7b:
Type: Secret (a volume populated by a Secret)
SecretName: cert-manager-token-9ck7b
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
kubectl describe clusterissuer
Name: letsencrypt-staging
Namespace:
Labels: <none>
Annotations: <none>
API Version: certmanager.k8s.io/v1alpha1
Kind: ClusterIssuer
Metadata:
Cluster Name:
Creation Timestamp: 2018-11-06T16:00:23Z
Generation: 1
Resource Version: 10184529
Self Link: /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-staging
UID: 11e44fe0-e1dd-11e8-8bc6-42010a840078
Spec:
Acme:
Email: dev@mydomain.com
Http 01:
Private Key Secret Ref:
Key:
Name: letsencrypt-staging
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Status:
Acme:
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/7297218
Conditions:
Last Transition Time: 2018-11-06T16:00:33Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
kubectl -n coffeer-ci describe certificate
Name: mydomain.fr
Namespace: coffeer-ci
Labels: <none>
Annotations: <none>
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Cluster Name:
Creation Timestamp: 2018-11-06T16:10:57Z
Generation: 1
Resource Version: 10197662
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/coffeer-ci/certificates/mydomain.fr
UID: 8b6d508a-e1de-11e8-8bc6-42010a840078
Spec:
Acme:
Config:
Domains:
mydomain.fr
Http 01:
Ingress: coffee-ingress
Common Name: mydomain.fr
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-staging
Secret Name: domain-tls
Status:
Acme:
Order:
Challenges:
Authz URL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/wm5MvoFA12U37qdXdBCccyIWezpEsLoxHUGVDacmHpI
Domain: mydomain.fr
Http 01:
Ingress: coffee-ingress
Key: RjHMkquS8Hh4dvJWZp2jLGW-MrSKEba-y8B8PzmVQ-M.4LwovuRj4ZgjrwLuye1cd5ftBRYaGIvtK__igMmDUD8
Token: RjHMkquS8Hh4dvJWZp2jLGW-MrSKEba-y8B8PzmVQ-M
Type: http-01
URL: https://acme-staging-v02.api.letsencrypt.org/acme/challenge/wm5MvoFA12U37qdXdBCccyIWezpEsLoxHUGVDacmHpI/192521366
Wildcard: false
URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/7297218/12596140
Conditions:
Last Transition Time: 2018-11-06T17:47:28Z
Message: http-01 self check failed for domain "mydomain.bap.fr"
Reason: ValidateError
Status: False
Type: Ready
Events: <none>
kubectl -n coffeer-ci describe ingress
Name: coffee-ingress
Namespace: coffeer-ci
Address: 35.233.8.223
Default backend: default-http-backend:80 (10.16.1.5:8080)
Rules:
Host Path Backends
---- ---- --------
mydomain.fr
/ coffee-service:80 (<none>)
/.well-known/acme-challenge/RjHMkquS8Hh4dvJWZp2jLGW-MrSKEba-y8B8PzmVQ-M cm-acme-http-solver-kw2w4:8089 (<none>)
Annotations:
ingress.kubernetes.io/forwarding-rule: k8s-fw-coffeer-ci-coffee-ingress--4b1e5690f5d3853f
ingress.kubernetes.io/target-proxy: k8s-tp-coffeer-ci-coffee-ingress--4b1e5690f5d3853f
ingress.kubernetes.io/url-map: k8s-um-coffeer-ci-coffee-ingress--4b1e5690f5d3853f
kubernetes.io/ingress.global-static-ip-name: coffeer-ci-static
kubernetes.io/tls-acme: true
ingress.kubernetes.io/backends: {"k8s-be-32603--4b1e5690f5d3853f":"HEALTHY"}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 40m nginx-ingress-controller Ingress coffeer-ci/coffee-ingress
Normal CreateCertificate 34m cert-manager Successfully created Certificate "domain-tls"
Warning Sync 25m (x23 over 59m) loadbalancer-controller Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce
Normal UPDATE 15m (x8 over 39m) nginx-ingress-controller Ingress coffeer-ci/coffee-ingress
Warning Sync 3m (x49 over 1h) loadbalancer-controller Error during sync: googleapi: Error 403: Quota 'BACKEND_SERVICES' exceeded. Limit: 9.0 globally., quotaExceeded
I have also the error Error 403: Quota 'BACKEND_SERVICES' exceeded. Limit: 9.0 globally., quotaExceeded in the ingress.
Thanks
Error preparing issuer for certificate coffeer-ci/mydomain.fr: http-01 self check failed for domain "mydomain.fr"
Means that it can't do the HTTP check that you actually own the domain. Do you own mydomain.fr
? If yes, you need to add a DNS entry to make mydomain.fr
resolve to the external IP (A record) of the load balancer (or if the load balancer has a name entry it would have to be a CNAME record, in the case of AWS ELBs) This way letsencrypt can use it to verify that you own the domain.
The other error:
Warning Sync 3m (x49 over 1h) loadbalancer-controller Error during sync: googleapi: Error 403: Quota 'BACKEND_SERVICES' exceeded. Limit: 9.0 globally., quotaExceeded
looks like the byproduct of not being able to verify the domain. If you don't specify and Ingress looks like cert-manager creates one for you with a 'LoadBalancer' type of service. It looks like it creates it initially but it keeps trying to sync to create it on GCP (maybe because to check if it can configure port 443) but after a while, the GCP API is throttling you.