K
Q

Horizontal Pod Autoscaler with custom metrics from Prometheus with percentiles for CPU usage

9/9/2019

So I am trying to figure out how can I configure an Horizontal Pod Autoscaler from a custom metric reading from Prometheus that returns CPU usage with percentile 0.95

I have everything set up to use custom metrics with prometheus-adapter, but I don't understand how to create the rule in Prometheus. For example, if I go to Grafana to check some of the Graphs that comes by default I see this metric:

sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace="api", pod_name="api-xxxxx9b-bdxx", container_name!="POD", cluster=""}) by (container_name)

But how can I modify that to be percentile 95? I tried with histogram_quantile function but it says no datapoints found:

histogram_quantile(0.95, sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace="api", pod_name="api-xxxxx9b-bdxx", container_name!="POD", cluster=""}) by (container_name))

But even if that works, will the pod name and namespace be filled by prometheus-adapter or prometheus when using custom metrics?

And every example I find using custom metrics are not related with CPU. So... other question I have is how people is using autoscaling metrics in production? I'm used to scale based on percentiles but I don't understand how is this managed in Kubernetes.

-- codiaf
kubernetes
prometheus

Similar Questions

Why would kubernetes OOM kill a pause container?
Unable to connect to ClusterIP service
how to block the option to `kubectl exec` and any other external connection to certain container on cluster
If crictl can pull an image from localhost, why does k3s fail to deploy it on localhost?
Getting Status code 422 when creating Kubernetes Job using nodejs

2 Answers

9/15/2019

If you want to add HPA based on custom metrics you can use Prometheus adapter.

Prometheus adapter helps you in exposing custom metrics to HPA.

Helm Chart - https://github.com/helm/charts/tree/master/stable/prometheus-adapter

Prometheus adapter - https://github.com/DirectXMan12/k8s-prometheus-adapter

Note - You have to enable 6443 port from public to cluster, because prometheus doesn’t provide override option.

https://github.com/helm/charts/blob/master/stable/prometheus-adapter/templates/custom-metrics-apiserver-deployment.yaml#L34

Make sure that Prometheus is getting custom metrics data Install Prometheus adapter on the same kubernetes cluster where you want apply hpa

helm install --name my-release stable/prometheus-adapter -f values.yaml

Pass the following config file to helm - values.yaml

prometheus-adapter:
  enabled: true
  prometheus:
    url: http://prometheus.namespace.svc.cluster.local
  rules:
    default: true
    custom:
    - seriesQuery: '{__name__="cpu",namespace!="",pod!="",service="svc_name"}'
      seriesFilters: []
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        matches: "cpu"
        as: "cpu_95"
      metricsQuery: "histogram_quantile(0.95, sum(irate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>,le))"

Above config will expose

cpu metrics as cpu_95 to HPA.

To Verify, if data is exposed properly run following command -

Fetch the data curl raw query command - kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/namespace_name/pods/\*/cpu_95 | jq .

HPA config -

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: test-cpu-manual
  labels:
    app: app_name
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta2
    kind: Deployment
    name: app_name
  minReplicas: 1
  maxReplicas: 15
  metrics:
  - type: Pods
    pods:
      metricName: cpu_95
      targetAverageValue: 75
-- Praveen_Shukla
Source: StackOverflow
9/10/2019

If I understand you correctly you don't have to use custom metrics in order to horizontally autoscale your pods. By default, you can automatically scale the number of Kubernetes pods based on the observed CPU utilization. Here is the official documentation with necessary details.

The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics).

The Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller. The resource determines the behavior of the controller. The controller periodically adjusts the number of replicas in a replication controller or deployment to match the observed average CPU utilization to the target specified by user.

And here you can find the walkthrough of how to set it up.

Also, here is the kubectl autoscale command documentation.

Example: kubectl autoscale rc foo --max=5 --cpu-percent=80

Auto scale a replication controller "foo", with the number of pods between 1 and 5, target CPU utilization at 80%

I believe that it is the easiest way so no need to complicate it with some custom metrics.

Please let me know if that helped.

-- OhHiMark
Source: StackOverflow