K
Q

Autoscaling Deployments with Cloud Monitoring metrics

4/26/2021

I am trying to auto-scale my pods based on CloudSQL instance response time. We are using cloudsql-proxy for secure connection. Deployed the Custom Metrics Adapter.

https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: application_name
spec:
  minReplicas: 1
  maxReplicas: 5
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: application_name
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metric:
        name: custom-metric-stackdriver-adapter
      target:
        type: AverageValue
        averageValue: 20

I deployed the application and create HPA for that, But i am seeing error.

AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric custom-metric: unable to fetch metrics from custom metrics API: the server could not find the descriptor for metric custom.googleapis.com/custom-metric: googleapi: Error 404: Could not find descriptor for metric 'custom.googleapis.com/custom-metric'., notFound
Events:
  Type     Reason               Age                      From                       Message
  ----     ------               ----                     ----                       -------
  Warning  FailedGetPodsMetric  4m22s (x10852 over 47h)  horizontal-pod-autoscaler  unable to get metric custom-metric: unable to fetch metrics from custom metrics API: the server could not find the descriptor for metric custom.googleapis.com/custom-metric: googleapi: Error 404: Could not find descriptor for metric 'custom.googleapis.com/custom-metric'., notFound
-- karthik
google-cloud-platform
google-cloud-stackdriver
google-kubernetes-engine
hpa
kubernetes

Similar Questions

How do I connect Power Bi to MongoDB running on Kubernetes with StatefulSets
Does Kubernetes Job/CronJob support diferent objects then Pods?
Kubernetes Ingress describe is not showing hostname
Are there any multi-host container orchestration tools which allow USB device mapping? Or any production grade solutions/hacks?
Kubernetes(minikube) + React Frontend + .netcore api + Cluster IP service + ingress + net::ERR_NAME_NOT_RESOLVED

1 Answer

5/21/2021
  1. Please refer to the link below to deploy a HorizontalPodAutoscaler (HPA) resource to scale your application based on Cloud Monitoring metrics.

https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#custom-metric_4

  1. Looks like the custom metric name is different in the app and hpa deployment configuration files(yaml). Metric and application names should be the same in both app and hpa deployment configuration files.

  2. In the hpa deployment yaml file,

    a. Replace custom-metric-stackdriver-adapter with custom-metric (Or change the metric name to custom-metric-stackdriver-adapter in the app deployment yaml file).

    b. Add “namespace: default” next to the application name at metadata.Also ensure you are adding the namespace in the app deployment configuration file.

    c. Delete the duplicate lines 6 & 7 (minReplicas: 1, maxReplicas: 5).

    d. Go to Cloud Console->Kubernetes Engine->Workloads. Delete the workloads (application-name & custom-metrics-stackdriver-adapter) created by app deployment yaml and adapter_new_resource_model.yaml files.

    e. Now apply configurations to resource model, app and hpa (yaml files).

-- Chandra Kiran Pasumarti
Source: StackOverflow