I'm running a 3 node k8s cluster where I have a prometheus instance that scrape my services metrics using object like serviceMonitor. In addition to my cluster services metrics, I would like to get metrics from an ambari-server. I know that it is possible to import ambari metrics as a datasource in grafana but I haven't seen anything about Prometheus. My goal would be to have those metrics in Prometheus to set alerts with AlertManager. I also saw that it is possible to write a go program that would expose metrics in the Prometheus format, but I preferred to ask to the community before trying stuffs.. :-)
If anyone had this issue, i would be happy if you cna help !
You can, Prometheus allows to scrape metrics exposed by external services: obviously they need to respect Prometheus format but I saw that it's possible.
You have to create static endpoints and related service:
apiVersion: v1
kind: Endpoints
metadata:
name: ambari-metrics
labels:
app: ambari
subsets:
- addresses:
- ip: "ambari-external-ip"
ports:
- name: metrics
port: 9100
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: ambari-metrics-svc
namespace: your-monitoring-namespace
labels:
app: ambari
spec:
type: ExternalName
externalName: ambari-external-ip
clusterIP: ""
ports:
- name: metrics
port: 9100
protocol: TCP
targetPort: 9100And finally, the ServiceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: ambari-metrics-sm
labels:
app: ambari
prometheus: kube-prometheus
spec:
selector:
matchLabels:
app: ambari
namespaceSelector:
matchNames:
your-monitoring-namespace
endpoints:
- port: metrics
interval: 30sObviously, YMMV and can find a detailed guide on this article