Prometheus allows me to dynamically load targets with file_sd_config from a .json file like this
#prometheus.yaml
- job_name: 'kube-metrics'
file_sd_configs:
- files:
- 'targets.json'[
{
"labels": {
"job": "kube-metrics"
},
"targets": [
"http://node1:8080",
"http://node2:8080"
]
}
]
However my targets differ in the metrics_path and not the host (I want to scrape metrics for every kubernetes node on <kube-api-server>/api/v1/nodes/<node-name>/proxy/metrics/cadvisor) but I can only set the metrics_path at the job level and not per target. Is this even achievable with prometheus or do I have to write my own code to scrape all these metrics and export them at a single target. Also I couldn't find a list of all supported auto discovery mechanisms, did I miss something in the docs?
You can use relabel_config in Prometheus config to change __metrics_path__ label config.
The principe is to provide the metrics path in your targets under the form host:port/path/of/metrics (note: drop the http://, it is in scheme parameter of scrape_config)
[
{
"targets": [
"node1:8080/first-metrics",
"node2:8080/second-metrics"
]
}
]
And then replace the related meta-labels with the parts
- job_name: 'kube-metrics'
file_sd_configs:
- files:
- 'targets.json'
relabel_configs:
- source_labels: [__address__]
regex: '[^/]+(/.*)' # capture '/...' part
target_label: __metrics_path__ # change metrics path
- source_labels: [__address__]
regex: '([^/]+)/.*' # capture host:port
target_label: __address__ # change target
You can reuse this method on any label known at configuration time to modify the config of the scrape.
On Prometheus, use the service discovery page to check your config has been correctly modified.
The official list of service discovery is in the configuration documentation: look for the *_sd_config in the index.