Specify Metricbeat Prometheus Field Names

9/19/2019

I am running my application and metricbeat on K8s. My application has an endpoint that exports prometheus metrics and metricbeat is able to pickup the metrics and add them to elastic.

The problem is that for a given document that makes it into elastic, the field name has the k8 namespace added to it. For example a given document will have a field named prometheus.my-namespace-1.runtime_memory_max_heap.value

This creates a problem since it means that each app that I add to my cluster will force a reindex on Kibana in order to detect the field type. Furthermore it means that I cannot make a general dashboard for my metrics and then filter on namespaces because each app has custom field names.

I would like the field name to just be called prometheus.runtime_memory_max_heap.value

-- ssc327
elasticsearch
kubernetes
metricbeat
prometheus

1 Answer

9/23/2019

As it turns out the issue was with my configuration. I had the following:

        config:
          - module: prometheus
            period: 15s
            metricsets: ["collector"]
            enabled: true
            hosts: ["${data.host}:9273"]
            ssl.verification_mode: "none"
            namespace: "${data.kubernetes.namespace}"

Once I updated the namespace it worked. I did try to remove namespace altogether but that did not work so I used a static value.

        config:
          - module: prometheus
            period: 15s
            metricsets: ["collector"]
            enabled: true
            hosts: ["${data.host}:9273"]
            ssl.verification_mode: "none"
            namespace: "metric"
-- ssc327
Source: StackOverflow