I am using prometheus and adapter to scale HPA (custom metrics memory_usage_bytes). I don't know why m is appended with targetValue and also HPA does not scaled down pods when they don't use memory.
Am i missing anything?
HPA code
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: pros
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: pros
maxReplicas: 3
metrics:
- type: Pods
pods:
metricName: memory_usage_bytes
targetAverageValue: 33000000kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
pros Deployment/pros 26781013333m/33M 1 3 3 19mcustom.metrics.k8.io
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/memory_usage_bytes"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "pros-6c9b9c5c59-57vmx",
"apiVersion": "/v1"
},
"metricName": "memory_usage_bytes",
"timestamp": "2019-07-13T12:03:10Z",
"value": "34947072",
"selector": null
},
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "pros-6c9b9c5c59-957zv",
"apiVersion": "/v1"
},
"metricName": "memory_usage_bytes",
"timestamp": "2019-07-13T12:03:10Z",
"value": "19591168",
"selector": null
},
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "pros-6c9b9c5c59-nczqq",
"apiVersion": "/v1"
},
"metricName": "memory_usage_bytes",
"timestamp": "2019-07-13T12:03:10Z",
"value": "19615744",
"selector": null
}
]
}
There are at least two good reasons explaining why it may not work:
As you can see in documentation:
The current stable version, which only includes support for CPU autoscaling, can be found in the autoscaling/v1 API version. The beta version, which includes support for scaling on memory and custom metrics, can be found in autoscaling/v2beta2.
and you are using:
apiVersion: autoscaling/v2beta1 in your HorizontalPodAutoscaler definition.
If you sum up total memory used by all 3 currently running pods from your custom.metrics.k8.io example, the workload still won't fit on just 2 Pods when memory limit is set to 33000000. Notice that first Pod have already reached its limit of 33M and memory consumption by other 2 Pods (19591168 + 19615744) is still too high to fit it on a single pod with the 3300000 limit.