Variable substitution in Prometheus yaml file

7/11/2019

I am deploying Prometheus with Thanos with the given config:

global:
  scrape_interval: 15s  # By default, scrape targets every 15 seconds.

 # Attach these labels to any time series or alerts when communicating with
 # external systems (federation, remote storage, Alertmanager).
 external_labels:
   monitor: %%CLUSTER%%
   replica: $(HOSTNAME)

I got this from the Thanos example documents but I am not able to make this work. "%%CLUSTER%% gives "cannot start with the given token" error whereas the HOSTNAME substitution never happens.

I read on this issue that variable substitution is not supported but the Thanos docs says otherwise. https://github.com/prometheus/prometheus/issues/2357

Is there any other way to implement this?

-- swetad90
kubernetes
prometheus
thanos
yaml

1 Answer

7/12/2019

UPDATE:

Looks like Thanos gives this out of the box. We have to configure Thanos to watch for the config file which has this templated variables and produce the output to a directory. And Prometheus then picks this file as its config file. This file would have the "substituted values" already from Thanos sidecar.

-- swetad90
Source: StackOverflow