Helm chart Prometheus unable to findTarget metrics placed in other namespace

4/30/2020

I got stuck at a very silly point. I created the cluster using kubeadm and using master node as worker node on my laptop. first i deployed the traefik ingress controller in 'kube-system' namespace and then php application in 'nirmal' namespace. Now i pulled the helm chart and modify according to my requirements. And i install the chart in monitoring namespace.

here is small part of my code of deployment of php app

php application deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-deploy
  namespace: nirmal
  labels:
    app.kubernetes.io/app: php-app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/app: php-app
  template:
    metadata:
      labels:
        app.kubernetes.io/app: php-app
    spec:
      containers:
        - name: php-app-container
          image: nirmalcontainer/php-mysql-image:v3
          # imagePullPolicy: Never
          ports:
            - containerPort: 80
          args:
            - --kubernetes
            - --logLevel=DEBUG
          # readinessProbe:
          #   httpGet:
          #     path: /
          #     port: 80
          #   initialDelaySeconds: 2
          #   periodSeconds: 2
      tolerations:
      - key: "key"
        value: "mosquito"
        effect: "NoSchedule" 

and the service for this deployment is

apiVersion: v1
kind: Service
metadata:
  name: php-svc
  namespace: nirmal
  labels:
    app.kubernetes.io/app: php-app
spec:
  selector:
    app.kubernetes.io/app: php-app
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http

And what i am doing in helm chart values.yaml file of prometheus chart is scrape_configs:

  - job_name: prometheus
    static_configs:
      - targets:
        - localhost:9090

  - job_name: php-svc.nirmal.svc.cluster.local
    scrape_interval: 5s
    static_configs:
      - targets: 
        - 10.104.207.188:80

  - job_name: mysql-svc.nirmal.svc.cluster.local
    scrape_interval: 5s
    static_configs:
      - targets: 
        - 10.105.69.96:3306

  - job_name: phpmyadmin-svc.nirmal.svc.cluster.local
    scrape_interval: 5s
    static_configs:
      - targets: 
        - 10.106.145.209:80

and the output is enter image description here

Now where i am wrong. please help me out.

-- NIRMAL SINGH
kubernetes
kubernetes-helm
php
prometheus

1 Answer

5/6/2020

Regarding the mysql-svc:

The transport connection broken: malformed HTTP response error means that you are trying to pass the request with a wrong protocol - "HTTP". In order to export mysql metrics you need to use an exporter, for example MySQL Server Exporter.

Regarding the other two:

  • Make sure you have /metrics endpoints implemented.

  • Check if loading http://10.104.207.188:80/metrics in your browser also returns 404. The same goes with the phpmyadmin. Make sure you use the right URL in your Prometheus config - metrics_path.

Please let me know if that helped.

-- OhHiMark
Source: StackOverflow