K
Q

Unable to find Prometheus custom app exporter as a target in Prometheus server deployed in Kubernetes

2/11/2021

I created a custom exporter in Python using prometheus-client package. Then created necessary artifacts to find the metric as a target in Prometheus deployed on Kubernetes.

But I am unable to see the metric as a target despite following all available instructions. Help in finding the problem is appreciated.

Here is the summary of what I did.

  1. Installed Prometheus using Helm on the K8s cluster in a namespace prometheus
  2. Created a python program with prometheus-client package to create a metric
  3. Created and deployed an image of the exporter in dockerhub
  4. Created a deployment against the metrics image, in a namespace prom-test
  5. Created a Service, ServiceMonitor, and a ServiceMonitorSelector
  6. Created a service account, role and binding to enable access to the end point

Following is the code.

Service & Deployment

apiVersion: v1
kind: Service
metadata:
  name: test-app-exporter
  namespace: prom-test
  labels:
    app: test-app-exporter
spec:
  type: ClusterIP
  selector:
    app: test-app-exporter
  ports:
  - name: http
    protocol: TCP
    port: 6000
    targetPort: 5000
    

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-exporter
  namespace: prom-test
spec:
  selector:
    matchLabels:
      app: test-app-exporter
  template:
    metadata:
      labels:
        app: test-app-exporter
    spec:
      #serviceAccount: test-app-exporter-sa
      containers:
      - name: test-app-exporter
        image: index.docker.io/cbotlagu/test-app-exporter:2
        imagePullPolicy: Always
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - name: http
          containerPort: 5000
      imagePullSecrets:
        - name: myregistrykey

Service account and role binding

apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-app-exporter-sa
  namespace: prom-test

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-app-exporter-binding
subjects:
  - kind: ServiceAccount
    name: test-app-exporter-sa
    namespace: prom-test
roleRef:
  kind: ClusterRole
  name: test-app-exporter-role
  apiGroup: rbac.authorization.k8s.io

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-app-exporter-role
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]

Service Monitor & Selector

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: test-app-exporter-sm
  namespace: prometheus
  labels:
    app: test-app-exporter
    release: prometheus
spec:
  selector:
    matchLabels:
      # Target app service
      app: test-app-exporter
  endpoints:
  - port: http
    interval: 15s
    path: /metrics
  namespaceSelector:
    matchNames:
    - default
    - prom-test
    #any: true
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: service-monitor-selector
  namespace: prometheus
spec:
  serviceAccountName: test-app-exporter-sa
  serviceMonitorSelector:
    matchLabels:
      app: test-app-exporter-sm
      release: prometheus
  resources:
    requests:
      memory: 400Mi
-- Padmaja
kubernetes
prometheus

Similar Questions

Spring Cloud Kubernetes: Cannot pass secrets into application_properties.yaml
How to install Helm Umbrella Charts pulled from remote repository
Containers versus Docker versus Kubernetes
How to handle ip address change in couchbase cluster?
Kubernetes mount volume on existing directory with files inside the container

1 Answer

2/14/2021

I am able to get the target identified by Prometheus. But though the end point can be reached within the cluster as well as from the node IP. Prometheus says the target is down. In addition to that I am unable to see any other target. Prom-UI

Any help is greatly appreciated

Following is my changed code Deployment & Service

apiVersion: v1
kind: Namespace
metadata:
  name: prom-test
---  
apiVersion: v1
kind: Service
metadata:
  name: test-app-exporter
  namespace: prom-test
  labels:
    app: test-app-exporter
spec:
  type: NodePort
  selector:
    app: test-app-exporter
  ports:
  - name: http
    protocol: TCP
    port: 5000
    targetPort: 5000
    

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-exporter
  namespace: prom-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app-exporter
  template:
    metadata:
      labels:
        app: test-app-exporter
    spec:
      serviceAccountName: rel-1-kube-prometheus-stac-operator
      containers:
      - name: test-app-exporter
        image: index.docker.io/cbotlagu/test-app-exporter:2
        imagePullPolicy: Always
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - name: http
          containerPort: 5000
      imagePullSecrets:
        - name: myregistrykey

Cluster Roles

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: prom-test
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  - endpoints
  - pods
  - services
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-from-prom-test
  namespace: prom-test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader
subjects:
- kind: ServiceAccount
  name: rel-1-kube-prometheus-stac-operator
  namespace: monitoring
---  
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: monitoring-role
  namespace: monitoring
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  - endpoints
  - pods
  - services
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-from-prom-test
  namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: monitoring-role
subjects:
- kind: ServiceAccount
  name: rel-1-kube-prometheus-stac-operator
  namespace: monitoring

Service Monitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: test-app-exporter-sm
  namespace: monitoring
  labels:
    app: test-app-exporter
    release: prometheus
spec:
  selector:
    matchLabels:
      # Target app service
      app: test-app-exporter
  endpoints:
  - port: http
    interval: 15s
    path: /metrics
  namespaceSelector:
    matchNames:
    - prom-test
    - monitoring
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: service-monitor-selector
  namespace: monitoring
spec:
  serviceAccountName: rel-1-kube-prometheus-stac-operator
  serviceMonitorSelector:
    matchLabels:
      app: test-app-exporter
      release: prometheus
  resources:
    requests:
      memory: 400Mi
-- Padmaja
Source: StackOverflow