Summary

I'm trying to figure out how to properly use the OR | operator in a Prometheus query because my imported Grafana dashboard is not working.

Long version

I'm trying to debug a Grafana dashboard based on some data scraped from my Kubernetes pods running AppMetrics/Prometheus; the dashboard is here. Basically what happens is that when the value "All" for the server is selected on the Grafana dashboard (server is an individual pod in this case), no data appears. However, when I select an individual pod, then data does appear.

Here's an example of the same metric scraped from the two pods:

# HELP application_httprequests_transactions 
# TYPE application_httprequests_transactions summary
application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 5.006965628
application_httprequests_transactions_count{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 1367
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.5"} 0.000202825
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.75"} 0.000279318
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.95"} 0.000329862
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.99"} 0.055584233

# HELP application_httprequests_transactions 
# TYPE application_httprequests_transactions summary
application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 6.10214788
application_httprequests_transactions_count{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 1363
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.5"} 0.000218548
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.75"} 0.000277483
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.95"} 0.033821094
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.99"} 0.097113234

I ran the Query inspector in Grafana to find out which query it is calling, and then ran the PromQL query in Prometheus itself. Basically, when I execute the following PromQL queries individually, they return data:

rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv"}[15m])*60

rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-jdq78"}[15m])*60

However, when I try to use PromQL's | operator to combine them, I don't get data back:

rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv|myapp-test-58d94bf78d-jdq78"}[15m])*60

Here's the raw output from Grafana's query inspector:

xhrStatus:"complete"

request:Object

method:"GET"
url:"api/datasources/proxy/56/api/v1/query_range?query=rate(application_httprequests_transactions_count%7Benv%3D%22test%22%2Capp%3D%22MyApp%22%2Cserver%3D%22myapp-test-58d94bf78d-jdq78%7Cmyapp-test-58d94bf78d-l9tdv%7Cmyapp-test-5b8c9845fb-7lklm%7Cmyapp-test-5b8c9845fb-8jf7n%7Cmyapp-test-5b8c9845fb-d9x5c%7Cmyapp-test-5b8c9845fb-fw4gj%7Cmyapp-test-5b8c9845fb-vtl9z%7Cmyapp-test-5b8c9845fb-vv7xv%7Cmyapp-test-5b8c9845fb-wq9bs%7Cmyapp-test-5b8c9845fb-xqfrt%7Cmyapp-test-69999d58b5-549vd%7Cmyapp-test-69999d58b5-lmp8x%7Cmyapp-test-69999d58b5-nbvt9%7Cmyapp-test-69999d58b5-qphj2%7Cmyapp-test-6b8dcc5ffb-gjjvj%7Cmyapp-test-6b8dcc5ffb-rxfk2%7Cmyapp-test-7fdf446767-bzhm2%7Cmyapp-test-7fdf446767-hp46w%7Cmyapp-test-7fdf446767-rhqhq%7Cmyapp-test-7fdf446767-wxmm2%22%7D%5B1m%5D)*60&start=1540574190&end=1540574505&step=15"

response:Object

status:"success"

data:Object

resultType:"matrix"

result:Array[0] => []

I opened a GitHub issue for this as well; it has a quick GIF screen recording showing what I mean: AppMetrics/Prometheus#43