Can someone guide the configuration for auto discover for K8s. The Prometheus server is outside of the cluster. I tried Service Discovery With Kubernetes and someone mentioned in this discussion
I'm not yet a K8s expert enough to explain all the details here, but fundamentally it's perfectly possible to run Prometheus outside of the cluster (and required for things like redundant cross-cluster meta-monitoring). Cf. the
in_clusterconfig option in http://prometheus.io/docs/operating/configuration/#kubernetes-sd-configurations-kubernetes_sd_config . You need to jump through certificate hoops if you run it outside.
So, I made a simple configuration
- job_name: 'kubernetes'
kubernetes_sd_configs:
-
# The API server addresses. In a cluster this will normally be
# `https://kubernetes.default.svc`. Supports multiple HA API servers.
api_servers:
- https://xxx.xx.xx.xx
# Run in cluster. This will use the automounted CA certificate and bearer
# token file at /var/run/secrets/kubernetes.io/serviceaccount/ in the pod.
in_cluster: false
# Optional HTTP basic authentication information.
basic_auth:
username: prometheus
password: secret
# Retry interval between watches if they disconnect.
retry_interval: 5s
Getting unknown fields in kubernetes_sd_config: api_servers, in_cluster, retry_interval" or some other indentation errors
In sample configuration, they mentioned ca_file:. How to get that certificate file from K8s or is there any way to specify K8s config file(~/.kube/config)
How to retrieve that file depends on your cluster setup
How to get that certificate file from K8s
By default, kubernetes stores the client CA certificate file at /etc/kubernetes/pki/ca.crt and also at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt in a ConfigMap used by kubeconfig.
By digging though the source code I figured out, that Prometheus always uses the in cluster config, if no api_server is provided in the config (discovery/kubernetes/kubernetes.go#L90-L96).
Somehow the docs don't say anything about the Kubernetes configuration parameters, but the source code does (config/config.go#L1026-L1037). Therefore there is not list named api_servers, but a single parameter named api_server.
So your config should look like this (untested):
- job_name: 'kubernetes'
kubernetes_sd_configs:
-
# The API server addresses. In a cluster this will normally be
# `https://kubernetes.default.svc`. Supports multiple HA API servers.
api_server: https://xxx.xx.xx.xx
# Optional HTTP basic authentication information.
basic_auth:
username: prometheus
password: secret
# specify the CA
tls_config:
ca_file: /path/to/ca.crt
## If the actual CA file isn't available you need to disable verification:
# insecure_skip_verify: true
I don't know where the retry_interval parameter comes from, but AFAIK this isn't a Kubernetes config parameter and it's also not part of the Prometheus config.
With help of @svenwltr answer I have create docker image which we can launch in K8s cluster. Check my repo