I have spring boot powered microservices deployed in my local kubernetes cluster. The microservices are using micrometer and prometheus registry but due to our company policy the actuator is available on another port:

• 8080 for "business" http requests
• 8081/manage for actuator. So, I can access http://host:8081/manage/prometheus and see the metrics when running the process locally (without kubernetes).

Now, I'm a beginner in Prometheus and have a rather limited knowledge in kubernetes (I'm coming with a Java developer background).

I've created a POD with my application and succesfully run it in kubernetes. It works and I can access it (for 8080 I've created a service to map the ports) and I can execute "business" level http requests it from the same PC.

But I haven't find any examples of adding a prometheus into the picture. Prometheus is supposed to be deployed in the same kubernetes cluster just as another pod. So I've started with:

FROM @docker.registry.address@/prom/prometheus:v2.15.2

COPY entrypoint.sh /
USER root
RUN chmod 755 /entrypoint.sh

ADD ./prometheus.yml  /etc/prometheus/

ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh looks like:

#!/bin/sh
echo "About to run prometheus"
/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
                --storage.tsdb.path=/prometheus \
                --storage.tsdb.retention.time=3d \
                --web.console.libraries=/etc/prometheus/console_libraries \
                --web.console.templates=/etc/prometheus/consoles

My question is about how exactly I should define prometheus.yml so that it will get the metrics from my spring boot pod (and other microservices that I have, all spring boot driven with the same actuator setup).

I've started with (prometheus.yml):

global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:

  - job_name: 'prometheus'
    metrics_path: /manage/prometheus
    kubernetes_sd_configs:
      - role: pod
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token  
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: sample-pod-app(.*)|another-pod-app(.*)

But apparently it doesn't work, so I've asking for the advices:

• If someone has a working example it would be the best :)
• Intuitively I understand I need to specify the port mapping for my 8081 port but I'm not exactly know how
• Since prometheus is supposed to run on another port am I supposed to expose a kubernetes service for port 8081 at the kubernetes level?
• Do I need any security related resources in kubernetes to be defined?

As a side note. At this point I don't care about scalability issues, I believe one prometheus server will do the job, but I'll have to add Grafana into the picture.