I am working on prometheus inside a Kubernetes enviornment, where I want to monitor our pods, which are being prepared to send metrics to prometheus directly. I am able to install prometheus via helm install stable/prometheus command, but the prometheus.yml scraping file is inside pods, and is also not persistent if pod restarts.

Since we are still experimenting, the scraping file will go through some iterations before we can say for sure this one works for us. Reason why I am sticking with helm is it also installs other packages like grafana, nodeexpoerter, etc, which are helpful

How can I instruct helm to use a specific data directory present on AWS. Let's say /var/prometheus. If that's not possible, then atleast a custom prometheus.yml, which when updated on server side can be reflected in prometheus pods.

As of now, atleast, I would like to add

        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - action: labeldrop
            regex: '(kubernetes_pod|app_kubernetes_io_instance|app_kubernetes_io_name|instance)'


This in the scraping file. What am I missing? Thank you. :-)