Custom prometheus query with k8s for HPA

2/19/2022

I want to do Horizontal pod autoscaling on the basis of input request. I made query for prometheus which returns aggregated input tps.

sum(sum by (path) (rate(dapr_http_server_request_count{app_id="governor",path=~"/v1.0/invoke/app/method/interceptor/.*"}[10s])))

I want to use this output in kubernetes HPA.

I am using prometheus-adapter for the same. Prometheus configuration is as follows:

 default: true
  custom:
    - seriesQuery: '{__name__=~"dapr_http_server_request_avg_.*",namespace!="",pod!=""}'
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: 'sum(sum by (path) (rate(dapr_http_server_request_count{app_id="governor",path=~"/v1.0/invoke/app/method/interceptor/.*"}[10s])))'

When i try to get the output in custom api of kubernetes it returns.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/app/pods/*/dapr_http_server_request_avg_total" | jq .

Error from server (NotFound): the server could not find the metric dapr_http_server_request_avg_total for pods
-- Uday Chauhan
hpa
kubernetes
prometheus

1 Answer

2/20/2022

You should check your prometheus-adapter config file, which makes me confused.

seriesQuery: dapr_http_server_request_avg_.*
metricsQuery: dapr_http_server_request_count

It is possible to get metrics dapr_http_server_request_count from above series?

      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"

and here take care, your metric name seems to be renamed, you should find the right metric name for you query.

try this: kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

you will see what your K8s Api-server actually get from Prometheus Adapter.

-- Leo Lao
Source: StackOverflow