I'm trying set up custom metrics with a HorizontalPodAutoscaler on a 1.6.1 alpha GKE cluster.
According to https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#prerequisites I need to set --horizontal-pod-autoscaler-use-rest-clients on kube-controller-manager to enable metrics collection. From GKE, it's not clear whether it's possible to set flags on kube-controller-manager. Any ideas?
Has anyone gotten custom metrics working with HPA on GKE?
On GKE we have been supporting HPA with custom metrics since version 1.9. If you have a group of horizontally autoscaled pods inside your cluster each exporting a custom metric then you can set an average per pod target for that metric.
An example of that would be an autoscaled deployment of a frontend where each replica exports its current QPS. One could set the average target of QPS per frontend pod and use the HPA to scale the deployment up and down accordingly. You can find the documentation and a tutorial explaining how to set this up here: https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling
Kubernetes 1.10 becoming available on GKE will extend the support for custom metrics to include metrics not attached to any Kubernetes object. This will give you the ability to scale a deployment based on any metric listed here, for example number of messages in Google Pub/Sub queue.
You can't manipulate any of the kubernetes cluster component directly in GKE(Google Container Engine), Google will do that job, if you want to achieve that you may need to deploy your own kubernetes cluster.