I configured automated SSL certificate management few months ago as described here: http://docs.cert-manager.io/en/latest/tutorials/acme/dns-validation.html for domains: <myhost>.com
and dev.<myhost>.com
. So I have two namespaces: prod
for <myhost>.com
and dev
for dev.<myhost>.com
. In each namespace I have ingress controller and Certificate
resource to store certificate to secret. It's working fine and ClusterIssuer
automatically updates certificates.
But few days ago I tried to add new domain: test.<myhost>.com
in test
namespace with absolutely same configuration of ingress and certificate as in prod
or dev
namespace (expect host name and namespace):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
kubernetes.io/tls-acme: 'true'
name: app-ingress
namespace: test
spec:
tls:
- hosts:
- test.<myhost>.com
secretName: letsencrypt-tls
rules:
- host: test.<myhost>.com
http:
paths:
- backend:
serviceName: web
servicePort: 80
path: /
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: cert-letsencrypt
namespace: test
spec:
secretName: letsencrypt-tls
issuerRef:
name: letsencrypt-prod-dns
kind: ClusterIssuer
commonName: 'test.<myhost>.com'
dnsNames:
- test.<myhost>.com
acme:
config:
- dns01:
provider: dns
domains:
- test.<myhost>.com
and this configuration doesnt work: certificate can't be found in secret, ingress is using "app-ingress-fake-certificate".
cert-manager
pod shows a lot of similar errors:
pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Challenge: challenges.certmanager.k8s.io is forbidden: User "system:serviceaccount:kube-system:cert-manager" cannot list challenges.certmanager.k8s.io at the cluster scope
pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Order: orders.certmanager.k8s.io is forbidden: User "system:serviceaccount:kube-system:cert-manager" cannot list orders.certmanager.k8s.io at the cluster scope
and certificate
is not trying to get certificate (kubectl describe -ntest cert-letsencrypt
):
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata: ...
Spec:
Acme:
Config:
Dns 01:
Provider: dns
Domains:
test.<myhost>.com
Common Name: test.<myhost>.com
Dns Names:
test.<myhost>.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-prod-dns
Secret Name: letsencrypt-tls
Events: <none>
It should have any status as certificates on other namespaces.
I can't understand why this configuration worked before but can't work now.
I'm not sure it's related, but I updated kubernetes using kops few weeks ago, current version is:
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"archive", BuildDate:"2018-10-12T16:56:06Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
The cause of this issue was Kubernetes upgrade from 1.9
to 1.10
. To fix it you need to upgrade cert-manager to 0.5.x
version.
It may be not possible to upgrade from 0.4.x
to 0.5.x
using helm
because of bug https://github.com/jetstack/cert-manager/issues/1134 in such case you need to store all issuers and certificates configurations then delete cert-manager 0.4.x
and install 0.5.x
, then apply all issuers and certificates configurations from first step.