K
Q

Question

Monitoring Kubernetes cluster using prometheus outside the k8 cluster

October 22, 2019

We have kubernetes cluster where I have service account "kube", namespace "monitoring" with cluster role binding created to monitor cluster
We have prometheus installed on a linux system (on prem) outside the cluster and is installed using "root"
When I try to connect to the k8 cluster with the https api using
```
ca.crt
```
and user
```
token
```
(given by kubernetes admin), it throws multiple errors.

Error messages:

component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig

component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig

Prometheus configuration:


  - job_name: 'kubernetes-apiservers'
    scheme: https
    tls_config:
      ca_file: /root/prometheus/ca.crt
    bearer_token_file: /root/prometheus/user_token
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://example.com:1234
      bearer_token_file: /root/prometheus/user_token
      tls_config:
        ca_file: /root/prometheus/prometheus-2.12.0.linux-amd64/ca.crt
    relabel_configs:
    - source_labels: [monitoring, monitoring-sa, 6443]
      action: keep
      regex: default;kubernetes;https
  
  - job_name: 'kubernetes-nodes'
    scheme: https
    tls_config:
        ca_file: /root/prometheus/ca.crt
    bearer_token_file: /root/prometheus/user_token
  
    kubernetes_sd_configs:
    - role: node
      api_server: https://example.com:1234
      bearer_token_file: /root/prometheus/user_token
      tls_config:
        ca_file: /root/prometheus/ca.crt
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: https://example.com:1234
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics

-- dmainmenu2

kubernetes

monitoring

prometheus

3 Answers

October 23, 2019

The main problem you're facing is:

"unable to use specified CA cert /root/prometheus/ca.crt"

Someone recently faced the same problem: https://github.com/prometheus/prometheus/issues/6015#issuecomment-532058465

He solved it by reinstalling the new version.

Version

2.13.1

is out. Try installing the latest version, it might solve your problem too.

-- Kamol Hasan

Source: StackOverflow

October 29, 2020

Your

ca.crt

is most probably still in

base64

format since secrets are encoded that way when describing them, as explained here.

-- Marko Stojakovic

Source: StackOverflow

November 7, 2019

Maybe your

ca.crt

have some error, check your ca cert file, make sure this file format like this:

-----BEGIN CERTIFICATE-----
xxxxx
-----END CERTIFICATE-----

I think your

ca.crt

is get by

kubectl get serviceaccount -o yaml

, but this is a public key with your kubernetes cluster, so, if you want to get the token, you can specify the

serviceAccountName

in the yaml file with a new

Deployment

, like this:

kind: Deployment

apiVersion: extensions/v1beta1

metadata:

  name: test

spec:

  replicas: 1

  template:

    metadata:

      labels:

        app: test

        version: v1

    spec:

      serviceAccountName: prometheus

      containers:
      - name: test

        image: alpine

        imagePullPolicy: Always

        command: ["ping", "127.0.0.1"]

      imagePullSecrets:
        - name: harbor-secret

      restartPolicy: Always

Then, get your

token

and

ca.crt

under

/var/run/secrets/kubernetes.io/serviceaccount/

.

-- Liu Yue

Source: StackOverflow

KQ

Monitoring Kubernetes cluster using prometheus outside the k8 cluster

Similar Questions

3 Answers

K
Q