I have a managed kubernetes setup, called Cluster Container Engine (CCE), in the Open Telekom Cloud. Their documentation can be found online.

My CCE has one master and three nodes which run k8s version 1.9.2 (more details below). I can access the CCE through kubectl and deploy new pods onto it.

The CCE has a deployment of heapster preinstalled. However, attempting to inspect node resource usage fails (I can observe the same effect for pod usage):

$ kubectl top pods
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)

I've attempted all debugging steps I could think of (see below) and I'm still lost when it comes to fixing this. Any advice?

The deployment, pod and service items for heapster are present (outputs filtered to include only heapster):

$ kubectl get po -n kube-system
NAME                                  READY   STATUS    RESTARTS   AGE
heapster-apiserver-84b844ffcf-lzh4b   1/1     Running   0          47m

$  kubectl get svc -n kube-system
NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                  AGE
heapster                 ClusterIP   10.247.150.244   <none>        80/TCP                                   19d

$ kubectl get deploy -n kube-system
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
heapster-apiserver   1/1     1            1           19d

To check that heapster does indeed collect metrics properly, I've ssh'd into one of the nodes and executed:

$ curl -k http://10.247.150.244:80/api/v1/model/metrics/              
[
  "cpu/usage_rate",
  "memory/usage",
  "cpu/request",
  "cpu/limit",
  "memory/request",
  "memory/limit"
 ]

Pod Log Output

Finally, I checked the log output from the heapster-apiserver-84b844ffcf-lzh4b pod:

$ kubectl logs -n kube-system heapster-apiserver-84b844ffcf-lzh4b
I0311 13:38:18.334525       1 heapster.go:78] /heapster --source=kubernetes.summary_api:''?kubeletHttps=true&inClusterConfig=false&insecure=true&auth=/srv/config --api-server --secure-port=6443
I0311 13:38:18.334718       1 heapster.go:79] Heapster version v1.5.3
I0311 13:38:18.340912       1 configs.go:61] Using Kubernetes client with master "https://192.168.1.228:5443" and version <nil>
I0311 13:38:18.340996       1 configs.go:62] Using kubelet port 10255
I0311 13:38:18.358918       1 heapster.go:202] Starting with Metric Sink
I0311 13:38:18.510751       1 serving.go:327] Generated self-signed cert (/var/run/kubernetes/apiserver.crt, /var/run/kubernetes/apiserver.key)
E0311 13:38:18.540860       1 heapster.go:128] Could not create the API server: missing clientCA file
I0311 13:38:18.558944       1 heapster.go:112] Starting heapster on port 8082

Cluster Info

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.2-CCE2.0.7-B003", GitCommit:"302f471a1e2caa114c9bb708c077fbb363aa2f13", GitTreeState:"clean", BuildDate:"2018-06-20T03:27:16Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get nodes
192.168.1.163   Ready    worker       19d   v1.9.2-CCE2.0.7-B003
192.168.1.211   Ready    nfs-server   19d   v1.9.2-CCE2.0.7-B003
192.168.1.227   Ready    worker       19d   v1.9.2-CCE2.0.7-B003

All nodes use EulerOS_2.0_SP2 with kernel version 3.10.0-327.59.59.46.h38.x86_64.