I've successfully deployed AKS with virtual nodes, where it automatically creates Azure Container Instances to support the number of pods requested, and I can manually scale up with:
kubectl scale --replicas=20 deployment.apps/blah
And sure enough I see 20 container instances get created in a special resource group and they're all running my app. When I scale it down, they disappear. This is great.
So then I try setting up autoscaling. I set limits/requests for CPU in my yaml and I say:
kubectl autoscale deployment blah --min=1 --max=20 --cpu-percent=50
But no new pods get created. To find out more I say:
kubectl describe hpa
And I get:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 3s (x12 over 2m49s) horizontal-pod-autoscaler unable to get metrics for resource cpu: no metrics returned from resource metrics API
Warning FailedComputeMetricsReplicas 3s (x12 over 2m49s) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
According to these docs the metrics-server
is automatically available in AKS since 1.8 (mine is 1.12, newly created).
Is this specifically a problem with virtual nodes, i.e. do they lack the ability to expose resource utilization via metrics-server
in the way required by autoscale
? Or is there something else I need to set up?
Metric-Server should be able to gather metrics from Virtual Kubelet (ACI)
Here's an example repo that shows that HPA with ACI is possible.