Is there a way to monitor tls certificates in kubernetes using prometheus?

8/1/2020

I want to monitor my tls certificates in Kubernetes using Prometheus and get a dashboard in grafana. I want to monitor their expiry and would want to get an alert when the certificates are going to be expired in 30 days. I did a lot of research and I finally found https://github.com/enix/x509-exporter. How do I use it? Is there any other efficient way to monitor the expiry of the certificates?

-- jyothi swarup
grafana
kubernetes
prometheus
ssl
tls1.2

3 Answers

8/1/2020

*DISCLAIMER: I haven't tried this x509-exporter. Just giving suggestion as per my understanding.*

The README file seems bit off. The first thing you need to do is create a github issue, no worries I raised one here.

I am listing down steps as per my understanding and referring the usage section.

  • Use their official docker image and deploy it as a deployment on k8s.
  • Check sample k8s yaml files for creating deployment. Also note that the deployment yaml should mount a host directory where all the k8s certificates are stored.
  • As per documentation, usually the certificates are located at /etc/kubernetes/pki.
  • The deployment yaml should contain a command where you point the exporter to the directory where certificates are located with other necessary options. Like this
    command: ["x509-exporter"]
    args: ["-d", "/etc/kubernetes/pki", "-p", "8091", "--debug"]
***Note:** Here I am running exporter in debug mode on port 8091, remember to expose this port.*
- In prometheus config, add the x509-exporter endpoint as target to scrape the metrics and plot those by creating graphs in Grafana dashboard.


  [1]: https://github.com/enix/x509-exporter
  [2]: https://github.com/enix/x509-exporter/issues/5
  [3]: https://github.com/enix/x509-exporter#usage
  [4]: https://hub.docker.com/r/enix/x509-exporter
  [5]: https://kubernetes.io/docs/setup/best-practices/certificates/#where-certificates-are-stored
  [6]: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#define-a-command-and-arguments-when-you-create-a-pod
-- mchawre
Source: StackOverflow

8/2/2020

Another way is to install the x509-exporter using the helm chart : https://hub.helm.sh/charts/enix/x509-exporter

See documentation here https://github.com/enix/helm-charts/tree/master/charts/x509-exporter.

You might also find the following prometheus alert rules useful (based on the x509-exporter metrics):

check-kubernetes-certificate.rules.yml :

groups:
- name: check-kubernetes-certificate-expiration.rules
  rules:
  - alert: KubernetesCertificateExpiration
    expr: floor((x509_cert_not_after - time()) / 86400) < 90
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: 'Certificate expiration on `{{ $labels.nb_cluster }}`'
      description: 'Certificate `{{ $labels.subject_CN }}` will expire in {{ $value }} days on `{{ $labels.nb_cluster }}`'

  - alert: KubernetesCertificateExpirationCritical
    expr: floor((x509_cert_not_after - time()) / 86400) < 10
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: 'Certificate expiration on `{{ $labels.nb_cluster }}`'
      description: 'Certificate `{{ $labels.subject_CN }}` will expire in {{ $value }} days on `{{ $labels.nb_cluster }}`'

  - alert: KubeletCertificateEmbedded
    expr: x509_cert_not_after{filename="kubelet.conf", embedded_kind="user"}
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: '{{ $labels.instance }}: Embedded certificate in {{ $labels.filename }}'
      description: '{{ $labels.nb_cluster }} has kubelet {{ $labels.subject_CN }} running with an embedded certificate in {{ $labels.filepath }}'
-- Romain Degez
Source: StackOverflow

9/27/2020

The official prometheus/blackbox_exporter have the ssl cert expiry info already.

 Name: "probe_ssl_earliest_cert_expiry",
 Help: "Returns earliest SSL cert expiry date",

So all you need is:

  1. Setup blackbox_exporter and the Probe rules to the domain you want to monitor.
    You can check my project kehao95/helm-prometheus-exporter to install blackbox_exporter via helm chart.
  2. config rule to monitor certificate expiring.

You can config your prometheusRule like this: (assuming you're using prometheus-operator)

rules: 
  - alert: TLS certificate expiring
    expr: (probe_ssl_earliest_cert_expiry - time())/86400 < 45
    labels:
      severity: warning
  - alert: TLS certificate expiring
    expr: (probe_ssl_earliest_cert_expiry - time())/86400 < 30
    labels:
      severity: critical
-- kehao
Source: StackOverflow