My Prometheus instance is currently deployed outside to the K8s cluster. I went through the below article : https://www.linkedin.com/pulse/monitoring-kubernetes-prometheus-outside-cluster-steven-acreman
and this gives a very good idea about accessing the K8s services via the api server. I am still trying to make it work (stuck with certificates issue).
However, my question is can't we use Prometheus to scape the ingress-controller hostname to scrape metrics endpoint. Or is that even a standard approach?
I could not find a single document or implementation around the same. Please suggest if anyone has implemented the above scenario or has any docs on it.
I would use Kubernetes SD configurations, and take adventage of roles to discover targets.
Kubernetes SD configurations allow retrieving scrape targets from Kubernetes' REST API and always staying synchronized with the cluster state.
You could use role type endpoints
:
The
endpoints
role discovers targets from listed endpoints of a service. For each endpoint address one target is discovered per port. If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.Available meta labels:
__meta_kubernetes_namespace
: The namespace of the endpoints object.__meta_kubernetes_endpoints_name
: The names of the endpoints object.- For all targets discovered directly from the endpoints list (those not additionally inferred from underlying pods), the following labels are attached:
__meta_kubernetes_endpoint_hostname
: Hostname of the endpoint.__meta_kubernetes_endpoint_node_name
: Name of the node hosting the endpoint.__meta_kubernetes_endpoint_ready
: Set totrue
orfalse
for the endpoint's ready state.__meta_kubernetes_endpoint_port_name
: Name of the endpoint port.__meta_kubernetes_endpoint_port_protocol
: Protocol of the endpoint port.__meta_kubernetes_endpoint_address_target_kind
: Kind of the endpoint address target.__meta_kubernetes_endpoint_address_target_name
: Name of the endpoint address target.- If the endpoints belong to a service, all labels of the
role: service
discovery are attached.- For all targets backed by a pod, all labels of the
role: pod
discovery are attached.
Or role type ingress
:
The
ingress
role discovers a target for each path of each ingress. This is generally useful for blackbox monitoring of an ingress. The address will be set to the host specified in the ingress spec.Available meta labels:
__meta_kubernetes_namespace
: The namespace of the ingress object.__meta_kubernetes_ingress_name
: The name of the ingress object.__meta_kubernetes_ingress_label_<labelname>
: Each label from the ingress object.__meta_kubernetes_ingress_labelpresent_<labelname>
:true
for each label from the ingress object.__meta_kubernetes_ingress_annotation_<annotationname>
: Each annotation from the ingress object.__meta_kubernetes_ingress_annotationpresent_<annotationname>
:true
for each annotation from the ingress object.__meta_kubernetes_ingress_scheme
: Protocol scheme of ingress,https
if TLS config is set. Defaults tohttp
.__meta_kubernetes_ingress_path
: Path from ingress spec. Defaults to/
.
If you wish to check detailed example of configuring Prometheus for Kubernetes please see this file.
Also you might be interested in 3rd party Prometheus Operator, which automates the setup of Peometheus on top of Kubernetes.