I've installed Prometheus and Grafana to monitor my K8S cluster and microservices using helm charts:

helm install monitoring prometheus-community/kube-promehteus-stack --values prometheus-values.yaml --version 16.10.0 --namespace monitoring --create-namespace

the content of promehteus-values.yaml is:

prometheus:
  prometheusSpec:
    serviceMonitorSelector:
      matchLabels:
        prometheus: devops
commonLabels:
  prometheus: devops
grafana:
  adminPassword: test123

Then I installed kong-ingress-controller using helm-charts:

helm install kong kong/kong --namespace kong --create-namespace --values kong.yaml --set ingressController.installCRDs=false

the content of kong.yaml file is:

podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8100"

I've also changed the value of metricsBindAdress in kube-proxy configmap to 0.0.0.0:10249 .

then I installed kong prometheus plugin using this yaml file :

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: prometheus
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
plugin: prometheus

My kong endpoint is :

$ kubectl edit endpoints -n kong kong-kong-proxy

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2021-10-27T03:28:25Z"
  creationTimestamp: "2021-10-26T04:44:57Z"
  labels:
    app.kubernetes.io/instance: kong
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: "2.6"
    enable-metrics: "true"
    helm.sh/chart: kong-2.5.0
  name: kong-kong-proxy
  namespace: kong
  resourceVersion: "6553623"
  uid: 91f2054f-7fb9-4d63-8b65-be098b8f6547
subsets:
- addresses:
  - ip: 10.233.96.41
    nodeName: node2
    targetRef:
      kind: Pod
      name: kong-kong-69fd7d7698-jjkj5
      namespace: kong
      resourceVersion: "6510442"
      uid: 26c6bdca-e9f1-4b32-91ff-0fadb6fce529
  ports:
  - name: kong-proxy
    port: 8000
    protocol: TCP
  - name: kong-proxy-tls
    port: 8443
    protocol: TCP

Finally I wrote the serviceMonitor for kong like this :

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  generation: 1
  labels:
    prometheus: devops
  name: kong-sm
  namespace: kong
spec:
  endpoints:
  - interval: 30s
    port: kong-proxy
    scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
    - kong
  selector:
    matchLabels:
      app.kubernetes.io/instance: kong

After all of this ; the targets in prometheus dashboard looks like this:

What did I miss/do wrong?