How can the Horizontal Pod Autoscaler be used to scale a custom resource (CR) based on the resource utilization of the child deployment?

8/21/2019

I have a custom resource that manages a deployment. I want my HPA to be able to scale the CR replica count based on the deployment CPU utilization instead of scaling the deployment directly. If it scales the deployment directly then when the reconciler loop is triggered it would just immediately see the discrepancy between the deployment replica count and the desired replica count as stated in the CR and update the deployment accordingly.

I am pretty close. I have the scale endpoint of my CR functioning properly and my HPA can even hit the endpoint. It just can't read the resource usage of the child.

I've also gotten it working if I have it scaling the deployment directly but as I stated above it's not a viable solution. More just proof that I have the metrics server functioning properly and the resource utilization is obtainable.

HPA YAML:

kind: HorizontalPodAutoscaler
metadata:
  name: {{.metadata.name}}
  namespace: {{.spec.namespace}}
spec:
  minReplicas: 1
  maxReplicas: 2
  metrics:
  - resource:
      name: cpu
      targetAverageUtilization: 2
    type: Resource
  scaleTargetRef:
    apiVersion: testcrds.group.test/v1alpha1
    kind: MyKind
    name: my-kind-1

And proof that the HPA is at least able to hit the scale endpoint of the CR:

Name:                                                  my-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           kubectl.kubernetes.io/last-applied-configuration:
                                                         {"kind":"HorizontalPodAutoscaler","apiVersion":"autoscaling/v2beta1","metadata":{"name":"my-kind-1","namespace":"default","creationTimestamp":n...
CreationTimestamp:                                     Wed, 21 Aug 2019 17:22:11 -0400
Reference:                                             MyKind/my-kind-1
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 2%
Min replicas:                                          1
Max replicas:                                          2
MLP pods:                                              0 current / 1 desired
Conditions:
  Type         Status  Reason            Message
  ----         ------  ------            -------
  AbleToScale  True    SucceededRescale  the HPA controller was able to update the target scale to 1
Events:
  Type    Reason             Age                   From                       Message
  ----    ------             ----                  ----                       -------
  Normal  SuccessfulRescale  3m54s (x80 over 23m)  horizontal-pod-autoscaler  New size: 1; reason: Current number of replicas below Spec.MinReplicas

As can be seen no dice on retrieving the resource utilization...

-- Scott Berman
kubernetes
kubernetes-operator

1 Answer

8/23/2019

I finally figured it out. Wrote a brief medium article as there are a few steps to follow, answer in article: https://medium.com/@thescott111/autoscaling-kubernetes-custom-resource-using-the-hpa-957d00bb7993

-- Scott Berman
Source: StackOverflow