How to track number of changes for Kubernetes HPA?

2/16/2022

Concerning the Kubernetes Horizontal Autoscaler, are there any metrics related to the number of changes between certain time periods?

-- Hegemon
hpa
kubernetes

1 Answer

2/23/2022

Kubernetes does not provide such metrics, but you can get events for a k8s resource.

An event in Kubernetes is an object in the framework that is automatically generated in response to changes with other resources—like nodes, pods, or containers.

The simplest way to get events for HPA:

$ kubectl get events | grep HorizontalPodAutoscaler
7m5s        Normal    SuccessfulRescale              HorizontalPodAutoscaler   New size: 8; reason: cpu resource utilization (percentage of request) above target
3m20s       Normal    SuccessfulRescale              HorizontalPodAutoscaler   New size: 10; reason:

or

$ kubectl describe hpa <yourHpaName>

But Kubernetes events are deleted by default after 1 hour (it is the default time-to-live, higher values might require more resources for etcd). Therefore you must watch for and collect important events as they happen.

To do this you can use for example:

  • KubeWatch is great open-source tool for watching and streaming K8s events to third-party tools and webhooks.
  • EventRouter is another great open-source tool for collecting Kubernetes events. It is effortless to set up and aims to stream Kubernetes events to multiple sources or sinks as they are referred to in its documentation. However, just like KubeWatch, it also does not offer querying or persistence features. You need to connect it with a third-party storage and analysis tool for a full-fledged experience.
  • almost any logging tool for k8s.

Also, you can use the official Kubernetes API library to develop some simple app for catching events from HPA. There is a good example of how to develop a simple monitoring app for k8s by yourself in this answer. They monitor pods' statuses there, but you can get a good idea of developing the app you need.

-- mozello
Source: StackOverflow