K
Q

How would I run a simple container the triggers Prometheus to reload its config when the underlying ConfigMap changes?

2/2/2017

While running Prometheus in Kubernetes, I'm pushing out new config via a ConfigMap. ConfigMaps are exposed as files in the container.

I would love Prometheus to automatically reload its configuration when the file changes.

Would something like this work?

inotifywait -q -m -e close_write /etc/prometheus/config.yml |
while read -r filename event; do
  curl -X POST http://localhost:9090/-/reload
done
-- checketts
kubernetes
prometheus

Similar Questions

No IPs in the GKE ca.cert which results in SSL errors while connection to the API server
Mounted ConfigMap not setting environment variables
What do hostPID and hostIPC options mean in a kubernetes pod?
Docker Container Usage

2 Answers

2/2/2017

Another option would be to use a livenessProbe command and just trigger a restart of the Pod when the configuration changed.

That could look like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        ports:
        - containerPort: 9090
        volumeMounts:
          - name: config-volume
            mountPath: /etc/prometheus
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - "test -z $(find /etc/prometheus -mmin -2)"
          initialDelaySeconds: 300
          periodSeconds: 10
      volumes:
        - name: config-volume
          configMap:
            name: prometheus

A drawback could be that this way you'd loose the data cached in memory, but it's straight forward and doesn't need a sidecar container.

-- pagid
Source: StackOverflow
2/2/2017

(Edit: I took some time to get this fully to work) this works with a small sidecar container. The configuration could look like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  template:
    ....
    spec:
      containers:
        ... (your actual container config goes here) ...
      - name: refresh
        imagePullPolicy: Always
        args:
        - /etc/prometheus/config.yml
        - http://localhost:9090/-/reload
        image: tolleiv/k8s-prometheus-reload
        volumeMounts:
          - name: config-volume
            mountPath: /etc/prometheus
      volumes:
        - name: config-volume
          configMap:
            name: prometheus

The actual check is done with this script where the observed file and the URL are passed as parameters:

#!/bin/sh
while true; do
   inotifywait "$(readlink -f $1)"
   echo "[$(date +%s)] Trigger refresh"
   curl -sSL -X POST "$2" > /dev/null
done

Everything can be found in this container on Dockerhub

Keeping a single inotifywait with the -m didn't work because of symlink juggling which is done by Kubernetes when the ConfigMap changes.

-- pagid
Source: StackOverflow