K
Q

Question

Prometheus Adapter Custom Metrics for Libvirt in a K8S Cluster

2/19/2019

I have a K8S cluster which is also managing VMs via virtlet. This K8S cluster is running K8S v1.13.2, with prometheus and the prometheus-adapter, and a custom-metrics server. I have written a custom metrics exporter for libvirtd which pulls in VM metrics and have configured prometheus to scrape that exporter for those VM metrics -- this is working and working well.

What I need to do next, is to have the prometheus-adapter push those metrics into K8S. Nothing I have done is working. Funny thing is, I can see the metrics in prometheus, but I am unable to present them to the custom metrics API.

Example metric visible in prometheus:

libvirt_cpu_stats_cpu_time_nanosecs{app="prometheus-lex",domain="virtlet-c91822c8-5e82-beta-deflect",instance="192.168.2.32:9177",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_pod_name="prometheus-lex-866694b884-9z8v6",name="prometheus-lex",pod_template_hash="866694b884"}

Prometheus Adapter configuration for this metric:

    - seriesQuery: 'libvirt_cpu_stats_cpu_time_nanosecs{job="kubernetes-pods", app="prometheus-lex"}'
      seriesFilters: []
      resource:
        overrides:
          kubernetes_pod_name:
            resource: pod
          kubernetes_namespace:
            resource: namespace
      name:
        matches: libvirt_cpu_stats_cpu_time_nanosecs
        as: libvirt_cpu_stats_cpu_time_rate
      metricsQuery: rate(libvirt_cpu_stats_cpu_time_nanosecs{job="kubernetes-pods", app="prometheus-lex", <<.LabelMatchers>>}[5m])

When I query the custom metrics API, I do not see what I am looking for:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1|grep libvirt

returns nothing

Additionally, I can see the prometheus-adapter is able to query the series from prometheus. So I know that side of the adapter is working. I am just trying to figure out why it's not presenting them to the custom metrics server.

From the prometheus-adapter

I0220 19:12:58.442937       1 api.go:74] GET http://prometheus-server.default.svc.cluster.local:80/api/v1/series?match%5B%5D=libvirt_cpu_stats_cpu_time_nanosecs%7Bkubernetes_namespace%21%3D%22%22%2Ckubernetes_pod_name%21%3D%22%22%7D&start=1550689948.392 200 OK

Any ideas what I am missing here?

Update::

I have also tried the following new configuration, and it's still not working.

- seriesQuery: 'libvirt_cpu_stats_cpu_time_nanosecs{kubernetes_namespace!="",kubernetes_pod_name!=""}'
  seriesFilters: []
  resource:
    overrides:
      kubernetes_namespace: {resource: "namespace"}
      kubernetes_pod_name: {resource: "pod"}
  name:
    matches: 'libvirt_cpu_stats_cpu_time_nanosecs'
    as: 'libvirt_cpu_stats_cpu_time_rate'
  metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

-- Rory Savage

kubernetes

libvirt

prometheus

1 Answer

7/10/2019

It actually depends on how you install the Prometheus Adapter. If you install via helm and use the YAML as configuration to the rules. You need to follow this README https://github.com/helm/charts/blob/master/stable/prometheus-adapter/README.md and and declare the rules like

rules:
  custom:
  - seriesQuery: '{__name__=~"^some_metric_count
quot;}'
    resources:
      template: <<.Resource>>
    name:
      matches: ""
      as: "my_custom_metric"
    metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)

Pay attention to the custom keyword. If you miss it, the number won't be available via custom metrics.

-- zhaocong

Source: StackOverflow

KQ

Prometheus Adapter Custom Metrics for Libvirt in a K8S Cluster

Similar Questions

1 Answer

K
Q