I'm trying to retrieve metrics from Spring Boot application in Openshift and I chose telegraf because it offers to transform metrics from Prometheus format to OpenTSDB which I need. The problem is I'm stuck at using a sidecar container of telegraf in each POD which is not the best stable solution. I need to create one POD of telegraf that would discover other services and other pods in order to retrieve metrics from them.

I added the role 'view' and 'hawkular-metrics' to default service account but it logs that it can't watch all the pods of the cluster and in fact, I'm trying to watch only the pods of one project and not all the projects of the cluster.

This is my template to deploy telegraf :

---
kind: Template
apiVersion: v1
metadata:
  name: telegraf-testapp-template
  annotations:
    description: Template to deploy an application with telegraf
labels:
  app: telegraf
  createdBy: telegraf-testapp-template
parameters:
  - description: Application name
    name: APPLICATION_NAME
    value: 'telegraf'
    required: true
  - description: OpenTSDB service
    name: OPENTSDB_HOST
    value: 'proxytsdb.project.svc'
  - description: OpenTSDB port
    name: OPENTSDB_PORT
    value: '4242'
  - description: Default data collection interval for telegraf (seconds)
    name: TELEGRAF_INTERVAL
    value: '1s'
  - description: Prefix for metrics keys
    name: PREFIX_KEYS
    value: 'app.'
  - description: Run telegraf with debug log messages (true/false)
    name: TELEGRAF_DEBUG
    value: 'false'
  - description: Cluster
    name: CLUSTER
    value: 'pmp'
  - description: Project name
    name: PROJECT
    value: 'unknown'
objects:
  - kind: DeploymentConfig
    apiVersion: v1
    metadata:
      name: ${APPLICATION_NAME}
      labels:
        deploymentConfig: ${APPLICATION_NAME}
        app: telegraf
    spec:
      replicas: 1
      selector:
        deploymentConfig: ${APPLICATION_NAME}
      strategy:
        type: Rolling
      template:
        metadata:
          labels:
            deploymentConfig: ${APPLICATION_NAME}
            app: telegraf
          name: ${APPLICATION_NAME}
        spec:
          containers:
            - env:
                - name: INTERVAL
                  value: ${TELEGRAF_INTERVAL}
                - name: PREFIX_KEYS
                  value: 'app.'
                - name: OPENTSDB_HOST
                  value: ${OPENTSDB_HOST}
                - name: OPENTSDB_PORT
                  value: ${OPENTSDB_PORT}
                - name: DEBUG
                  value: ${TELEGRAF_DEBUG}
                - name: CLUSTER
                  value: ${CLUSTER}
                - name: PROJECT
                  value: ${PROJECT}
                - name: SERVICE
                  value: ${APPLICATION_NAME}
              image: >-
                sandbox/awl-openshift-telegraf:1.9.1
              imagePullPolicy: Always
              name: telegraf
              resources: {}
              securityContext: {}
              terminationMessagePath: /dev/termination-log
        restartPolicy: Always
        dnsPolicy: ClusterFirst
      triggers:
        - type: ConfigChange

After deploying this template with the service account default, I expect that it would discover the pods and services of the project but it shows :

unable to watch resources: kubernetes api: Failure 403 User "system:serviceaccount:project:default" cannot watch all pods in the cluster