I want to monitor my tls certificates in Kubernetes using Prometheus and get a dashboard in grafana. I want to monitor their expiry and would want to get an alert when the certificates are going to be expired in 30 days. I did a lot of research and I finally found https://github.com/enix/x509-exporter. How do I use it? Is there any other efficient way to monitor the expiry of the certificates?
*DISCLAIMER: I haven't tried this x509-exporter. Just giving suggestion as per my understanding.*
The README file seems bit off. The first thing you need to do is create a github issue, no worries I raised one here.
I am listing down steps as per my understanding and referring the usage section.
/etc/kubernetes/pki
.command: ["x509-exporter"]
args: ["-d", "/etc/kubernetes/pki", "-p", "8091", "--debug"]
***Note:** Here I am running exporter in debug mode on port 8091, remember to expose this port.*
- In prometheus config, add the x509-exporter endpoint as target to scrape the metrics and plot those by creating graphs in Grafana dashboard.
[1]: https://github.com/enix/x509-exporter
[2]: https://github.com/enix/x509-exporter/issues/5
[3]: https://github.com/enix/x509-exporter#usage
[4]: https://hub.docker.com/r/enix/x509-exporter
[5]: https://kubernetes.io/docs/setup/best-practices/certificates/#where-certificates-are-stored
[6]: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#define-a-command-and-arguments-when-you-create-a-pod
Another way is to install the x509-exporter using the helm chart : https://hub.helm.sh/charts/enix/x509-exporter
See documentation here https://github.com/enix/helm-charts/tree/master/charts/x509-exporter.
You might also find the following prometheus alert rules useful (based on the x509-exporter metrics):
check-kubernetes-certificate.rules.yml :
groups:
- name: check-kubernetes-certificate-expiration.rules
rules:
- alert: KubernetesCertificateExpiration
expr: floor((x509_cert_not_after - time()) / 86400) < 90
for: 5m
labels:
severity: warning
annotations:
summary: 'Certificate expiration on `{{ $labels.nb_cluster }}`'
description: 'Certificate `{{ $labels.subject_CN }}` will expire in {{ $value }} days on `{{ $labels.nb_cluster }}`'
- alert: KubernetesCertificateExpirationCritical
expr: floor((x509_cert_not_after - time()) / 86400) < 10
for: 5m
labels:
severity: critical
annotations:
summary: 'Certificate expiration on `{{ $labels.nb_cluster }}`'
description: 'Certificate `{{ $labels.subject_CN }}` will expire in {{ $value }} days on `{{ $labels.nb_cluster }}`'
- alert: KubeletCertificateEmbedded
expr: x509_cert_not_after{filename="kubelet.conf", embedded_kind="user"}
for: 5m
labels:
severity: warning
annotations:
summary: '{{ $labels.instance }}: Embedded certificate in {{ $labels.filename }}'
description: '{{ $labels.nb_cluster }} has kubelet {{ $labels.subject_CN }} running with an embedded certificate in {{ $labels.filepath }}'
The official prometheus/blackbox_exporter have the ssl cert expiry info already.
Name: "probe_ssl_earliest_cert_expiry",
Help: "Returns earliest SSL cert expiry date",
So all you need is:
You can config your prometheusRule
like this: (assuming you're using prometheus-operator)
rules:
- alert: TLS certificate expiring
expr: (probe_ssl_earliest_cert_expiry - time())/86400 < 45
labels:
severity: warning
- alert: TLS certificate expiring
expr: (probe_ssl_earliest_cert_expiry - time())/86400 < 30
labels:
severity: critical