I have a Kubernetes cluster on AWS, which was launched with Kops. I think initially launched as v1.7.0 but have updated to v1.9.10 since.

All running fine, except horizontal pod autoscaling not working.

Error has switched between the following 2, depending on what config I have set.

"unable to get metrics for resource cpu: no metrics returned from heapster"

"unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)"

I am aware that earlier versions defaulted to using heapster, while newer versions use metrics API.

I have tried installing metrics and heapster, both on their own and together. With heapster installed I see extra graphs in the dashboard so looks to be working fine.

spec:
  kubeControllerManager:
    horizontalPodAutoscalerUseRestClients: true

Most discussions on this issue bring up this config above, which have tried both true and false, in all combinations of heapster vs metrics setup.

kube-controller-manager 1.8.* had --horizontal-pod-autoscaler-use-rest-clients not set by default, whereas 1.9.* has it set as true by default. So my upgrade path suggest this could well be related.

Tried a load of other suggestions from SO and google, but can't recall all of them.

Below is YAML from the HPA, not sure what other outputs will be helpful in diagnosing this issue but will update with extra if needed.

{
  "kind": "HorizontalPodAutoscaler",
  "apiVersion": "autoscaling/v1",
  "metadata": {
    "name": "myappname-mobile-deployment",
    "namespace": "myappname",
    "selfLink": "/apis/autoscaling/v1/namespaces/myappname/horizontalpodautoscalers/myappname-mobile-deployment",
    "uid": "a8cd45d0-02a4-11e9-9660-12306a211636",
    "resourceVersion": "53939792",
    "creationTimestamp": "2018-12-18T09:09:44Z",
    "annotations": {
      "autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2018-12-18T09:10:14Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2018-12-18T09:10:14Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from heapster\"}]"
    }
  },
  "spec": {
    "scaleTargetRef": {
      "kind": "Deployment",
      "name": "myappname-mobile-deployment",
      "apiVersion": "extensions/v1beta1"
    },
    "minReplicas": 1,
    "maxReplicas": 5,
    "targetCPUUtilizationPercentage": 30
  },
  "status": {
    "currentReplicas": 1,
    "desiredReplicas": 0
  }
}