Kubernetes Autoscaling additional custom metrics

9/18/2019

I want to add custom metrics to my existing cpu metric, so I want two metrics. My second metrics has to be a custom metric/external metrics which makes a request to a webserver and gets there a value, is this possible?

At the moment it looks like this but I want to add a second metric but how?

  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

As I read in the docs kubernetes would use the higher metric to scale thats ok. Does someone have an example for me how to apply this custom metric in my case?

-- Java
kubernetes

1 Answer

9/18/2019

If it's an external metric (i.e. a custom metric that is not associated with a Kubernetes object):

 metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80
  - type: External
    external:
      metric:
        name: your_metric
      target:
        type: Value
        value: "100"

Note that in this case, the HPA will try to query the your_metric metric from the External Metrics API. That means, this API must exist in your cluster for this configuration to work.

If the metric is associated with a Kubernetes object, you would use a type: Object and if the metric is from the pods of the pod controller (e.g. Deployment) that you are trying to autoscale, you would use a type: Pods. In both cases, the HPA would try to fetch the metric from the Custom Metrics API.


Note (because it seems that you are trying to use a metric that is not yet in Kubernetes):

The HPA can only talk to the metric APIs: Resource Metrics API, Custom Metrics API, External Metrics API.

If your metric is not served by one of these APIs, then you have to create a metrics pipeline that brings the metric to one of these APIs.

For example, using Prometheus and the Prometheus Adapter:

  • Prometheus periodically scrapes the metric from your external web server
  • Prometheus Adapter exposes the metric through the External Metrics API

EDIT: explain metric APIs.

In the diagrams below, the green components are those that you need to install to provide the corresponding metric API.

Resource Metrics API

Serves CPU and memory usage metrics of all Pods and Nodes in the cluster. These are predefined metric (in contrast to the custom metrics of the other two APIs).

The raw data for the metrics is collected by cAdvisor which runs as part of the kubelet on each node. The metrics are exposed by the Metrics Server.

The Metrics Server implements the Resource Metrics API. It is not installed by default in Kubernetes. That means, to enable the Resource Metrics API in your cluster, you have to install the Metrics Server.

enter image description here

Custom Metrics API

Serves custom metrics that are associated with Kubernetes objects. The metrics can be anything you want.

You are responsible yourself for collecting the metrics that you want to expose through the Custom Metrics API. You do this by installing a "metrics pipeline" in the cluster.

You can choose the components for your metrics pipeline yourself. The only requirement is that the metrics pipeline is able to:

  1. Collect metrics
  2. Implement the Custom Metrics API

A popular choice for a metrics pipeline is to use Prometheus and the Prometheus Adapter:

  • Prometheus collects metrics (any metrics you want)
  • Prometheus Adapter implements the Custom Metrics API and exposes the metrics collected by Prometheus through the Custom Metrics API

enter image description here

External Metrics API

Serves custom metrics that are not associated with Kubernetes objects.

The External Metrics API works identical to the Custom Metrics API. The only difference is that it has different API paths (that don't include objects, but only metric names).

To provide the External Metrics API you can, in most cases, use the same metrics pipeline as for the Custom Metrics API (e.g. Prometheus and the Prometheus Adapter).

enter image description here

-- weibeld
Source: StackOverflow