Prometheus Adapter Goroutine issue when fetch the query from external.api

10/5/2019

I have implemented and registered external api to the k8s environment but I have some issues when fetch the metrics .

That is rule for external api metrics :

  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
    resources:
      overrides:
        kubernetes_namespace: {resource: "namespace"}
        kubernetes_pod_name: {resource: "pod"}
    name:
      matches: "^(.*)_total"
      as: "${1}"

When check the kubernetes external api metrics , resources values is not null that looks fine but ,

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "http_requests",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

After all those things, when I want check the value of http_requests metric kubernetes returning this error ;

Error from server (InternalError): Internal error occurred: unable to fetch metrics

I have checked this issue from prometheus-adapter pod these are error logs related with this query ;

E1005 01:21:26.387384       1 provider.go:47] unable to generate a query for the metric: empty query produced by metrics query template
I1005 01:21:26.387594       1 wrap.go:42] GET /apis/external.metrics.k8s.io/v1beta1/namespaces/default/http_requests: (504.956µs) 500
goroutine 1909 [running]:
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420374930, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420374930, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42105b2a0, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc42107f920, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:307 +0x45
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x1791e20, 0x10, 0x7f340078e6c8, 0xc4204ec580, 0x190fd20, 0xc420e8db88, 0xc421081600, 0x1f4, 0x18f9920, 0xc420039320)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d

....
....
....

But I have parsed this error message : unable to generate a query for the metric: empty query produced by metrics query template

What is the reason of this issue ?

Help please

-- Ayhan Balik
grafana
kubernetes
monitoring
prometheus

1 Answer

10/16/2019

I think that you are receiving this error:

unable to generate a query for the metric: empty query produced by metrics query template

because your api metric do not contain metricsQuery value.

Second thing is that I am not sure why you are using external.metrics in http_request. In that scenario you can use custom.metrics. Mentioned metrics is one of the most popular so you can find many tutorials online step-by-step how to achieve this. For example this or this Github tutorial.

-- PjoterS
Source: StackOverflow