I am deploying Prometheus with Thanos with the given config:
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: %%CLUSTER%%
replica: $(HOSTNAME)
I got this from the Thanos example documents but I am not able to make this work. "%%CLUSTER%% gives "cannot start with the given token" error whereas the HOSTNAME substitution never happens.
I read on this issue that variable substitution is not supported but the Thanos docs says otherwise. https://github.com/prometheus/prometheus/issues/2357
Is there any other way to implement this?
UPDATE:
Looks like Thanos gives this out of the box. We have to configure Thanos to watch for the config file which has this templated variables and produce the output to a directory. And Prometheus then picks this file as its config file. This file would have the "substituted values" already from Thanos sidecar.