Kubernetes - HPA replicas issue

12/10/2019
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myservice
  namespace: mynamespace
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myservice
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 700Mi

After 15 min, i see REPLICAS grown up to 2. Since then it's not coming down.

C:\Users\myuser>kubectl get hpa
NAME            REFERENCE                  TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
myservice    Deployment/myservice    490412032/700Mi          1         3         2          4h14m

There is no traffic to these services, as shown below MEMORY is well below 700Mi (as mentioned in HPA metrics).

C:\Users\myuser>kubectl top pod
NAME                                     CPU(cores)   MEMORY(bytes)
myservice-6ff6bdc8d-jx4pc             29m          463Mi
myservice-6ff6bdc8d-zktnm             29m          471Mi

I am puzzled why the 2nd replica has got created and why it's not going down ? Am i missingl something ? please suggest.

-- Molay
kubernetes

1 Answer

12/10/2019

The reason it's not going down:

The way the HPA controller calculates the number of replicas is

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

In your case the currentMetricValue is calculated from the average of the given metric across the pods, so (463 + 471)/2 = 467Mi because of the targetAverageValue being set.

The desiredMetricValue is 700Mi, so the desiredReplicas are

ceil[2 * (467 / 700)] = ceil[1.34] = 2

as ceil(x) will give you the smallest integer that is greater or equal to x.


The reason it went up:

Assuming the first pod was at about 463Mi, the desiredReplicas are

ceil[2 * (463 / 700)] = ceil[1.32] = 2

So if you want it to be only one pod for that workload, the targetAbsoluteValue needs to be at least 950Mi.

-- char
Source: StackOverflow