I have implemented and registered external api to the k8s environment but I have some issues when fetch the metrics .
That is rule for external api metrics :
- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "^(.*)_total"
as: "${1}"
When check the kubernetes external api metrics , resources values is not null that looks fine but ,
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "http_requests",
"singularName": "",
"namespaced": true,
"kind": "ExternalMetricValueList",
"verbs": [
"get"
]
}
]
}
After all those things, when I want check the value of http_requests
metric kubernetes returning this error ;
Error from server (InternalError): Internal error occurred: unable to fetch metrics
I have checked this issue from prometheus-adapter
pod these are error logs related with this query ;
E1005 01:21:26.387384 1 provider.go:47] unable to generate a query for the metric: empty query produced by metrics query template
I1005 01:21:26.387594 1 wrap.go:42] GET /apis/external.metrics.k8s.io/v1beta1/namespaces/default/http_requests: (504.956µs) 500
goroutine 1909 [running]:
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420374930, 0x1f4)
/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420374930, 0x1f4)
/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42105b2a0, 0x1f4)
/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc42107f920, 0x1f4)
/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:307 +0x45
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x1791e20, 0x10, 0x7f340078e6c8, 0xc4204ec580, 0x190fd20, 0xc420e8db88, 0xc421081600, 0x1f4, 0x18f9920, 0xc420039320)
/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d
....
....
....
But I have parsed this error message : unable to generate a query for the metric: empty query produced by metrics query template
What is the reason of this issue ?
Help please
I think that you are receiving this error:
unable to generate a query for the metric: empty query produced by metrics query template
because your api metric do not contain metricsQuery
value.
Second thing is that I am not sure why you are using external.metrics
in http_request
. In that scenario you can use custom.metrics
. Mentioned metrics is one of the most popular so you can find many tutorials online step-by-step how to achieve this. For example this or this Github tutorial.