First off a little new to using helm...
So I'm struggling to get the helm deployment of this: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
To work the way I would like in my kubernetes cluster. I like what it has done so far but how can I make it scrape a custom endpoint? I have seen this: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus
Under the section titled: "Scraping Pod Metrics via Annotations". I have added the following annotations to the pod deployment (and then the node port service) in kubernetes:
annotations = {
"prometheus.io/scrape" = "true"
"prometheus.io/path" = "/appMetrics/prometheusMetrics"
"prometheus.io/port" = "443"
}
However, when I look in the targets page of prometheus I don't see it there. I also don't see it in the configuration file. So that makes me think this helm chart isn't deploying the same prometheus chart.
So now the question is, how can I setup a custom scrape endpoint using the helm chart kube-prometheus-stack. From my reading this is the one I should* be using, right?
Try this below in your custom_values.yaml and apply it.
prometheus:
prometheusSpec:
additionalScrapeConfigs:
- job_name: your_job_name
scrape_interval: 15s
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- your_namespace
relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__address__]
action: replace
regex: ([^:]+)(?::\d+)?
replacement: ${1}:your_port
target_label: __address__
- source_labels: [__meta_kubernetes_pod_label_app]
action: keep
regex: your_pod_nameYou need to replace your_job_name, your_namespace, your_port, your_pod_name to your deployment file. After I did the above metric and re-install Prometheus by helm chart, now I can see the target, and the metrics get exposed.