GCP kubernetes HPA not working as described

11/21/2017

We have been running our workload on kubernetes on GCP for around a year now, however, last week, one of our websites was hit hard by a promo campaign launched by the site owner but without us prescaling the cluster.

Since then we've been load testing the stack on a test cluster, the issue we're having is that using a Google node, the HPA doesn't scale up as written in the documentation, rather it always scales

2 --> 4 -->8 -->16 -->32 --64--128

Regardless of CPU load.

For instance in our test we had 251%/60% cpu with 8 pods running. My math would suggest (251/60)*8= 33 pods are needed, however, it will always go to 16 next, then wait 3 mins before going to 32.

The scale out required for the test to function is around 64 pods, which it arrives at after about 25 mins, instead of 8, is there a way to get the GCP version operating more like the manual?

    apiVersion: autoscaling/v1
    kind: HorizontalPodAutoscaler
    metadata:
      annotations:
        autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"False","lastTransitionTime":"2017-11-        22T14:17:35Z","reason":"BackoffBoth","message":"the
          time since the previous scale is still within both the downscale and upscale
          forbidden windows"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2017-11-21T19:09:34Z","reason":"ValidMetricFound","message":"the
          HPA was able to succesfully calculate a replica count from cpu resource utilization
          (percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2017-11-22T14:11:05Z","reason":"DesiredWithinRange","message":"the
          desired replica count is within the acceptible range"}]'
        autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":64,"currentAverageValue":"193m"}}]'
      creationTimestamp: 2017-11-20T15:44:48Z
      name: varnish-7
      namespace: default
      resourceVersion: "373498"
      selfLink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/varnish-7
      uid: bd60211b-ce09-11e7-af0d-42010a8e0099
    spec:
      maxReplicas: 60
      minReplicas: 2
      scaleTargetRef:
        apiVersion: extensions/v1beta1
        kind: Deployment
        name: varnish-7
      targetCPUUtilizationPercentage: 50
    status:
      currentCPUUtilizationPercentage: 64
      currentReplicas: 8
      desiredReplicas: 8
      lastScaleTime: 2017-11-22T14:16:05Z
-- Rob Davis
gcp
google-cloud-platform
google-kubernetes-engine
kubernetes

0 Answers