Export kubernetes custom metrics with Prometheus for all namespaces

8/21/2020

I have a k8s cluster if prometheus-operator (namespace monitoring) and prometheus-adapter installed using helm charts. I can scrape metrics from other namespace (say rabbitmq) with prometheus-operator and prometheus-adapter can get custom metrics:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/rabbitmq/services/rabbitmq-server/rabbitmq_queue_messages" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/rabbitmq/services/rabbitmq-server/rabbitmq_queue_messages"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Service",
        "namespace": "rabbitmq",
        "name": "rabbitmq-server",
        "apiVersion": "/v1"
      },
      "metricName": "rabbitmq_queue_messages",
      "timestamp": "2020-08-20T12:15:39Z",
      "value": "0",
      "selector": null
    }
  ]
}

Here's my prometheus-adapter config:

prometheus:
  url: http://pmt-server-prometheus-oper-prometheus.monitoring.svc
  port: 9090

rbac:
  create: true

serviceAccount:
  create: true

service:
  port: 443

logLevel: 6
rules:
  custom:

  - seriesQuery: 'rabbitmq_queue_messages{namespace!="",service!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        service: {resource: "service"}
    metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,queue="task_queue"}) by (<<.GroupBy>>)

Problem is that those metrics are namespaces bounded, so I can only use this metric inside the same namespace rabbitmq. Is there a way to export those metrics to all namespaces so I can scale deployments in others namespaces (namespace default, for example)? In other words, make the metrics not namespace bound?

-- Toledo Neto
kubernetes
metrics
prometheus
prometheus-operator

0 Answers