K
Q

Hpa not fetching existing custom metric?

9/29/2019

I'm using mongodb-exporter for store/query the metrics via prometheus. I have set up a custom metric server and storing values for that .

That is the evidence of prometheus-exporter and custom-metric-server works compatible .

Query:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/*/mongodb_mongod_wiredtiger_cache_bytes"

Result:

{"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/%2A/mongodb_mongod_wiredtiger_cache_bytes"},"items":[{"describedObject":{"kind":"Pod","namespace":"monitoring","name":"mongo-exporter-2-prometheus-mongodb-exporter-68f95fd65d-dvptr","apiVersion":"/v1"},"metricName":"mongodb_mongod_wiredtiger_cache_bytes","timestamp":"TTTTT","value":"0"}]}

In my case when I create a hpa for this custom metrics from mongo exporter, hpa return this error to me :

failed to get mongodb_mongod_wiredtiger_cache_bytes utilization: unable to get metrics for resource mongodb_mongod_wiredtiger_cache_bytes: no metrics returned from resource metrics API

What is the main issue on my case ? I have checked all configs and flow is looking fine, but where is the my mistake .

Help

Thanks :)

-- Ayhan Balik
autoscaling
grafana
kubernetes
monitoring
prometheus

Similar Questions

Attach NGINX Configurations files with Persistence Volume on AWS EKS
Application load balancer distributing between elastic beanstalk and kubernetes
Job status is marked as Running when the pod is still pending for being scheduled
How to get clients public IP on the pod?
Unable to open service via kubectl proxy
Kubernetes NodePort connection refused

1 Answer

10/14/2019

In comments you wrote that you have enabled external.metrics, however in original question you had issues with custom.metrics

In short:

  • metrics supports only basic metric like CPU or Memory.
  • custom.metrics allows you to extend basic metrics to all Kubernetes objects (http_requests, number of pods, etc.).
  • external.metrics allows to gather metrics which are not Kubernetes objects:

External metrics allow you to autoscale your cluster based on any metric available in your monitoring system. Just provide a metric block with a name and selector, as above, and use the External metric type instead of Object

For more detailed description, please check this doc.

Minikube

To verify if custom.metrics are enabled you need to execute command below and check if you can see any metrics-server... pod.

$ kubectl get pods -n kube-system
...
metrics-server-587f876775-9qrtc    1/1     Running   4          5d1h

Second way is to check if minikube have enabled metrics-server by

$ minikube addons list
...
- metrics-server: enabled

If it is disabled just execute

$ sudo minikube addons enable metrics-server
✅  metrics-server was successfully enabled

GKE

Currently at GKE heapster and metrics-server are turn on as default but custom.metrics are not supported by default. You have to install prometheus adapter or stackdriver.

Kubeadm

Kubeadm do not include heapster or metrics server at the beginning. For easy installation, you can use this YAML.

Later you have to install prometheus adapter.

Apply custom.metrics

It's the same for Minikube, Kubeadm, GKE.

Easiest way to apply custom.metrics is to install prometheus adapter via Helm.

After helm installation you will be able to see note:

NOTES:
my-release-prometheus-adapter has been deployed.
In a few minutes you should be able to list metrics using the following command(s):

  kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

As additional information, you can use jq to get more user friendly output.

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .

-- PjoterS
Source: StackOverflow