K
Q

Retrieve custom `custom.metrics.k8s.io` value using curl

2/19/2019

I can list all the custom.metrics available, but I don't know how to query an individual value. For example I have tried:

curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/ | jq .

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "deployments.apps/aws_sqs_approximate_number_of_messages_visible_average",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

But if I try this:

curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/deployments.apps/aws_sqs_approximate_number_of_messages_visible_average | jq .

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "the server could not find the requested resource",
  "reason": "NotFound",
  "details": {

  },
  "code": 404
}

I get a 404. I've seen this issue which shows how to get a namespaced metric, but mine does not have a namespace? Is there a definition for how to use this API?

-- Breedly
kubernetes

Similar Questions

Why do certain kube-system Pods such as kube-proxy have the same Pod IP as the node that they are on?
Kubernetes service not accessible from out of network
Jenkins X per environment values.yaml
What is the best practice for deploying Prometheus monitoring Kubernetes?
MongoDB ignores authentication when called from inside Kubernetes cluster but not from outside

2 Answers

2/19/2019

Since I'm using DirectXMan12/k8s-prometheus-adapter there are a few things to know:

  1. I think it can only work with namespaced metrics.
  2. If a query does not return a metric for a particular time period in prometheus k8s-prometheus-adapter will report it as non-existent.
    • This is my actual problem.

Using the custom metrics API is very simple:

  1. kubectl proxy to open a proxy to your kubernetes API
  2. curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/ to list all custom metrics available.

For example you may see:

    {
      "name": "deployments.extensions/kube_deployment_status_replicas_available",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }

We know it is namespaced from namespaced: true and beneath the namespace we can select via deployment from the name field.

So we would build our query like so:

curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/namespace/$NAMESPACE/deployments.extensions/$DEPLOYMENT/kube_deployment_status_replicas_available

At least I think that's how it should work, although if you do the same query without deployments.extensions section it will show the value for the namespace:

curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/namespace/$NAMESPACE/kube_deployment_status_replicas_available

Perhaps this is due to how the query executes in prometheus.

-- Breedly
Source: StackOverflow
2/20/2019

Just like Resource Metrics, Custom Metrics are bound to Kubernetes objects too. What you're missing in your URL is the resource you want the metric to relate to. For example the Pod the custom metric is related to, but the same is true for Deployments.

Try to adjust this url to your needs:

kubectl get --raw \
'/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pod/podinfo-67c9fd95d-fqk4g/http_requests_per_second' \
| jq .

Here are the slides for the talk we gave at FOSDEM 2019 on the Prometheus Adapter: https://speakerdeck.com/metalmatze/kubernetes-metrics-api?slide=26

I'll update this answer, once the video is available too.

-- Matthias Loibl
Source: StackOverflow