I have a case where I want to set up an alert where at least one value of the label is distinct.
For example, a Kubernetes cluster xyz (having 20 nodes) with metric test_metric{cluster_name="xyz",os="ubuntu"}. I want to find out/setup an alert if any of these 20 nodes are having different "os" values.
Basically, the idea is to get an alert when the os's value is not the same across all nodes in a cluster.
At the moment I am testing a very simple rule which I think is not correct:
count(test_metric{cluster_name="xyz",os!=""} != count(test_metric{cluster_name="xyz",os!=""})Nested counts is the way to handle this:
count by (cluster_name) (
count by (os, cluster_name)(test_metric)
) != 1