Horizontal Pod Autoscaler (HPA) on Google Kubernetes Engine (GKE) using Backend Latency from an Ingress LoadBalancer via Stackdriver External Metric

3/20/2019

I'm trying to configure a Horizontal Pod Autoscaler (HPA) on Google Kubernetes Engine (GKE) using External Metrics from an Ingress LoadBalancer, basing the configuration on instructions such as

https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling and https://blog.doit-intl.com/autoscaling-k8s-hpa-with-google-http-s-load-balancer-rps-stackdriver-metric-92db0a28e1ea

With an HPA like

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: my-api
  namespace: production
spec:
  minReplicas: 1
  maxReplicas: 20
  metrics:
  - external:
      metricName: loadbalancing.googleapis.com|https|request_count
      metricSelector:
        matchLabels:
          resource.labels.forwarding_rule_name: k8s-fws-production-lb-my-api--63e2a8ddaae70
      targetAverageValue: "1"
    type: External
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-api

the autoscaler does kick in when the request count rises - but placing heavy load on the service, like 100 simultaneous requests per second, doesn't increase the external metric request_count much beyond 6 RPS, while the observed backend_latencies metric in Stackdriver does increase significantly; so I'd like to utilise that metric by adding to the HPA configuration, like so:

  - external:
      metricName: loadbalancing.googleapis.com|https|backend_latencies
      metricSelector:
        matchLabels:
          resource.labels.forwarding_rule_name: k8s-fws-production-lb-my-api--63e2a8ddaae70
      targetValue: "3000"
    type: External

but that results in the error:

...unable to fetch metrics from external metrics API: googleapi: Error 400: Field aggregation.perSeriesAligner had an invalid value of "ALIGN_RATE": The aligner cannot be applied to metrics with kind DELTA and value type DISTRIBUTION., badRequest

which can be observed with the command

$ kubectl describe hpa -n production

or by visiting

http://localhost:8080/apis/external.metrics.k8s.io/v1beta1/namespaces/default/loadbalancing.googleapis.com%7Chttps%7Cbackend_latencies

after setting up a proxy with

$ kubectl proxy --port=8080

Are https/backend_latencies or https/total_latencies not supported as External Stackdriver Metrics in an HPA configuration for GKE?

-- Bjorn Thor Jonsson
google-cloud-platform
google-kubernetes-engine
kubernetes

0 Answers