We have been running our workload on kubernetes on GCP for around a year now, however, last week, one of our websites was hit hard by a promo campaign launched by the site owner but without us prescaling the cluster.
Since then we've been load testing the stack on a test cluster, the issue we're having is that using a Google node, the HPA doesn't scale up as written in the documentation, rather it always scales
2 --> 4 -->8 -->16 -->32 --64--128
Regardless of CPU load.
For instance in our test we had 251%/60% cpu with 8 pods running. My math would suggest (251/60)*8= 33 pods are needed, however, it will always go to 16 next, then wait 3 mins before going to 32.
The scale out required for the test to function is around 64 pods, which it arrives at after about 25 mins, instead of 8, is there a way to get the GCP version operating more like the manual?
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
annotations:
autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"False","lastTransitionTime":"2017-11- 22T14:17:35Z","reason":"BackoffBoth","message":"the
time since the previous scale is still within both the downscale and upscale
forbidden windows"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2017-11-21T19:09:34Z","reason":"ValidMetricFound","message":"the
HPA was able to succesfully calculate a replica count from cpu resource utilization
(percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2017-11-22T14:11:05Z","reason":"DesiredWithinRange","message":"the
desired replica count is within the acceptible range"}]'
autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":64,"currentAverageValue":"193m"}}]'
creationTimestamp: 2017-11-20T15:44:48Z
name: varnish-7
namespace: default
resourceVersion: "373498"
selfLink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/varnish-7
uid: bd60211b-ce09-11e7-af0d-42010a8e0099
spec:
maxReplicas: 60
minReplicas: 2
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: varnish-7
targetCPUUtilizationPercentage: 50
status:
currentCPUUtilizationPercentage: 64
currentReplicas: 8
desiredReplicas: 8
lastScaleTime: 2017-11-22T14:16:05Z