According to this page https://cloud.google.com/compute/docs/autoscaler/#specifications : "Do not use Compute Engine autoscaling with managed instance groups that are owned by Google Kubernetes Engine". Any idea why ? I am doing this in production and that works well.
Most of my K8s resources are daemonset, each time a new K8s node pops with the managed IG autoscaler, some pods are scheduled on that node and the load is effectively balanced.
I find running HPA with Node Autoscaler to be cumbersome. We have to wait for both to kick in to actually scale, set resources requests precisely and run Heapster to measure actual resources consumption. Managed IG autoscaler is much simpler and reactive.
I've read https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#should-i-use-a-cpu-usage-based-node-autoscaler-with-kubernetes but there's not much information besides "don't do it"/"Not compatible". Sure "a node that has some system-critical pods on it" could be deleted but that's why we are running 3 copies of such system-critical pods.
So question is, do you see any other reason why this would not work ?
Using Compute Engine (GCE) autoscaling with Kubernetes (GKE), you can get into a situation where GKE wants to schedule a new pod but there's no node available because your GCE autoscaling trigger hasn't been hit (e.g. nodes are under the CPU threshold). Using GKE autoscaling, in contrast, will create a new node if it needs one in order to schedule a pod. It removes the heuristic guesswork of autoscaling in GCE.
That said, I've found that there are times where GCE autoscaling is still appropriate. I use two different node pools, one with Kubernetes autoscaling for my "regular" pods and one with GCE autoscaling for the pods that need to scale differently. I use nodeSelector to tell pods which node pool they should run on.