Kubernetes HPA incorrectly calculating averageUtilization for CPU

2/22/2022

I am setting up nginx autoscaling based on CPU.

Setup on the nginx deployment is:

        resources:
          limits:
            cpu: "2"
            memory: 1000Mi
          requests:
            cpu: 100m
            memory: 100Mi

When I check kubectl top pod

I see I have 4 pods. Each pod is using 2m. So that is 8m total. Can someone explain to me how when I check the HPA it shows 44%/50% is utilized? That math is definitely wrong.

HPA config:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: api-svc-nginx-ingress
  namespace: api-svc
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-ingress
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

If I manually scale down the deployment to 1 pod I see that the utilization drops to 8%/50%... Not sure whats wrong? Is metric server broken?

-- alex
hpa
kubernetes
kubernetes-ingress

1 Answer

2/22/2022

So it turns out I was using an additional metric that wasn't working (it was a custom metric that didn't have a source from metric server) - and I think that broke the metrics from updating.

I had another

  metrics:
  - type: Object

that was showing up as <unknown> on the HPA targets output. Once I removed that metric the HPA seemed to work fine.

-- alex
Source: StackOverflow