I'm trying to configure a Horizontal Pod Autoscaler (HPA) on Google Kubernetes Engine (GKE) using External Metrics from an Ingress LoadBalancer, basing the configuration on instructions such as
https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling and https://blog.doit-intl.com/autoscaling-k8s-hpa-with-google-http-s-load-balancer-rps-stackdriver-metric-92db0a28e1ea
With an HPA like
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: my-api
namespace: production
spec:
minReplicas: 1
maxReplicas: 20
metrics:
- external:
metricName: loadbalancing.googleapis.com|https|request_count
metricSelector:
matchLabels:
resource.labels.forwarding_rule_name: k8s-fws-production-lb-my-api--63e2a8ddaae70
targetAverageValue: "1"
type: External
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-api
the autoscaler does kick in when the request count rises - but placing heavy load on the service, like 100 simultaneous requests per second, doesn't increase the external metric request_count
much beyond 6 RPS, while the observed backend_latencies
metric in Stackdriver does increase significantly; so I'd like to utilise that metric by adding to the HPA configuration, like so:
- external:
metricName: loadbalancing.googleapis.com|https|backend_latencies
metricSelector:
matchLabels:
resource.labels.forwarding_rule_name: k8s-fws-production-lb-my-api--63e2a8ddaae70
targetValue: "3000"
type: External
but that results in the error:
...unable to fetch metrics from external metrics API: googleapi: Error 400: Field aggregation.perSeriesAligner had an invalid value of "ALIGN_RATE": The aligner cannot be applied to metrics with kind DELTA and value type DISTRIBUTION., badRequest
which can be observed with the command
$ kubectl describe hpa -n production
or by visiting
after setting up a proxy with
$ kubectl proxy --port=8080
Are https/backend_latencies
or https/total_latencies
not supported as External Stackdriver Metrics in an HPA configuration for GKE?