I'm trying to configure Prometheus outside Kubernetes Cluster.
Below is my Prometheus config.
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
api_server: https://10.0.4.155:6443
scheme: https
tls_config:
insecure_skip_verify: true
basic_auth:
username: kube
password: Superkube01
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;httpsThese is how it looks:
root@master01:~# kubectl cluster-info
Kubernetes master is running at https://10.0.4.155:6443
root@master01:~# kubectl get endpoints
NAME ENDPOINTS AGE
kubernetes 10.0.4.103:6443,10.0.4.138:6443,10.0.4.155:6443 11h
netchecker-service 10.2.0.10:8081 11h
root@master01:~# But, when starting Prometheus, i'm getting below error.
level=error ts=2018-05-29T13:55:08.171451623Z caller=main.go:216 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:270: Failed to list *v1.Pod: Get https://10.0.4.155:6443/api/v1/pods?resourceVersion=0: x509: certificate signed by unknown authority"Could anyone please tell me, what wrong i'm doing here?
Thanks, Pavanasam R
The error indicates that Prometheus is using a different certificate to sign its metric collection request than the one expected by your apiserver.
You really need to format your code in a code block so we can see the yaml formatting. kubernetes_sd_configs seems to be the wrong home for insecure_skip_verify and basic_auth according to this link. Might want to move them and try scraping again.
As of now your insecure_skip_verify is a part of kubernetes_sd_configs:. Add it in api_server context as well.
kubernetes_sd_configs:
- api_server: https://<ip>:6443
role: node
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
insecure_skip_verify: true