I'm using Grafana plus Prometheus queries to create dashboards in Grafana for Kubernetes. I take the name of the nodes (3 in this case) in a variable and then I pass this values to other query to extract the IPs of the machines. The values extracted are correct. I have the multi-value option enabled.
The problem comes with the query sum(rate(container_cpu_usage_seconds_total{id="/", instance=~"$ip_test:10250"}[1m])) and more than one IP because it only takes one of them. In other query it works but I think it is possible because the other query has not the :10250 after the variable.
My question, do you know any way to concatenate all the ip:port? E.g.: X.X.X.X:pppp|X.X.X.X:pppp
From multiple values formating documentation, Prometheus variables are expanded as regex:
InfluxDB and Prometheus uses regex expressions, so the same variable would be interpolated as (host1|host2|host3). Every value would also be regex escaped if not, a value with a regex control character would break the regex expression.
Therefore your variable ip_test = ['127.0.0.1', '127.0.0.2',...] is supposed to be transformed into: (127\.0\.0\.1|127\.0\.0\.2).
This means that your expression =~$ip_test:10250 should be transformed into =~"(127\.0\.0\.1|127\.0\.0\.2):10250" so you don't need the multiple expansion you are asking for.
The reason it is not working is that either the documentation is incorrect or there is a bug in Grafana (tested with version v6.7.2). From my tests, I suspect, the Prometheus expansion doesn't expand with the enclosing () and you end up with the expression =~"127\.0\.0\.1|127\.0\.0\.2:10250" - which is not what you want.
The workaround is to use the regex notation =~"${ip_test:regex}:10250".