I am trying to expose some custom metrics of a kubernetes application at prometheus.

I successfully deploy deploy my app at kubernetes. The ServiceMonitor is also added but no targets are discovered (0/0 up) . The application is an nginx server with the relevant nginx-prometheus-exporter sidecar.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-example-v3
  labels:
    app: nginx-example-v3
spec:
  selector:
    matchLabels:
      app: nginx-example-v3
  template:
    metadata:
      labels:
        app: nginx-example-v3
    spec:
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            memory: "128Mi"
            cpu: "100m"
        ports:
        - name: http
          containerPort: 8080
        volumeMounts:
        - name: "config"
          mountPath: "/etc/nginx/nginx.conf"
          subPath: "nginx.conf"
      - name: exporter
        image: nginx/nginx-prometheus-exporter:0.8.0
        ports:
        - containerPort: 9113
      volumes:
        - name: "config"
          configMap:
            name: "nginx-example-v2-config"
            
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: nginx-example-v3
  name: nginx-example-v3
spec:
  type: LoadBalancer
  selector:
    app: nginx-example-v3
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: http-exporter
    port: 9113
    targetPort: 9113

After that i can see the nginx custom metrics exposed at the /metrics API:

Then i apply the monitoring service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-example-v3
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 15s
    port: web
  selector:
    matchLabels:
      app: nginx-example-v3

i can see that the service is successfully added at prometheus "Service Discovery" section:

BUT no targets are discovered (0/0 up) on behalf of prometheus

What i am missing??? Any help is really apreciated since i have been stuck on this many days! Thank you very much in advance. :-)

Replies @efotopoulou