Azure Kubernetes with virtual nodes and autoscale - is this possible?

6/2/2019

I've successfully deployed AKS with virtual nodes, where it automatically creates Azure Container Instances to support the number of pods requested, and I can manually scale up with:

kubectl scale --replicas=20 deployment.apps/blah

And sure enough I see 20 container instances get created in a special resource group and they're all running my app. When I scale it down, they disappear. This is great.

So then I try setting up autoscaling. I set limits/requests for CPU in my yaml and I say:

kubectl autoscale deployment blah --min=1 --max=20 --cpu-percent=50

But no new pods get created. To find out more I say:

kubectl describe hpa

And I get:

  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedGetResourceMetric       3s (x12 over 2m49s)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  3s (x12 over 2m49s)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

According to these docs the metrics-server is automatically available in AKS since 1.8 (mine is 1.12, newly created).

Is this specifically a problem with virtual nodes, i.e. do they lack the ability to expose resource utilization via metrics-server in the way required by autoscale? Or is there something else I need to set up?

-- Daniel Earwicker
azure-aks
azure-kubernetes

1 Answer

3/24/2020

Metric-Server should be able to gather metrics from Virtual Kubelet (ACI)

Here's an example repo that shows that HPA with ACI is possible.

https://github.com/Azure-Samples/virtual-node-autoscale

-- Karishma Tiwari - MSFT
Source: StackOverflow