First of all I'd like to understand clearly something, if I run in a kubernetes cluster a telegraf daemonset, it will collect the metrics of the pods? Or it will collect the metrics of the physical nodes?

I've created a telegraf daemonset in my test kubernetes cluster running on my laptop under hyperv, based on this kubernetes cluster installation:

I would like to collect metrics of the pods but it doesn't arrive to the kafka machine. I get this error in the logs:

2019-05-08T02:36:35Z I! Starting Telegraf 1.9.2
2019-05-08T02:36:35Z I! Using config file: /etc/telegraf/telegraf.conf
2019-05-08T02:46:36Z E! [agent] Failed to connect to output kafka, retrying in 15s, error was 'kafka: client has run out of available brokers to talk to (Is your cluster reachable?)'

This is the daemonset definition file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: telegraf
  namespace: monitoring
  labels:
    k8s-app: telegraf
data:
  telegraf.conf: |+
    [global_tags]
      env = "$ENV"
    [agent]
      hostname = "$HOSTNAME"
      interval = "60s"
      round_interval = true
      metric_batch_size = 1000
      metric_buffer_limit = 10000
      collection_jitter = "0s"
      flush_interval = "10s"
      flush_jitter = "2s"
      precision = ""
      debug = false
      quiet = true
      logfile = ""

    [[outputs.kafka]]
      brokers = ["10.121.63.5:9092", "10.121.63.18:9092", "10.121.62.64:9092", "10.121.62.80:9092", "10.121.63.22:9092"]
      topic = "telegraf-measurements-json"
      client_id = "golangsarama__1.18.0__serverinfra__telegraf"
      routing_tag = "host"
      version = "0.11.0.2"
      compression_codec = 2
      required_acks = 1
      data_format = "json"

    [[inputs.cpu]]
      percpu = true
      totalcpu = true
      collect_cpu_time = false
      report_active = false
    [[inputs.disk]]
      ignore_fs = ["tmpfs", "devtmpfs", "devfs"]
    [[inputs.diskio]]
    [[inputs.kernel]]
    [[inputs.mem]]
    [[inputs.processes]]
    [[inputs.swap]]
    [[inputs.system]]
    [[inputs.docker]]
      endpoint = "unix:///var/run/docker.sock"
    [[inputs.kubernetes]]
      url = "https://192.168.213.18:6443"
      insecure_skip_verify = true

---
# Section: Daemonset
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: telegraf
  namespace: monitoring
  labels:
    k8s-app: telegraf
spec:
  selector:
    matchLabels:
      name: telegraf
  template:
    metadata:
      labels:
        name: telegraf
    spec:
      containers:
      - name: telegraf
        image: docker.io/telegraf:1.9.2
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 500m
            memory: 500Mi
        env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: "HOST_PROC"
          value: "/rootfs/proc"
        - name: "HOST_SYS"
          value: "/rootfs/sys"
        - name: ENV
          valueFrom:
            secretKeyRef:
              name: telegraf
              key: env
        volumeMounts:
        - name: sys
          mountPath: /rootfs/sys
          readOnly: true
        - name: proc
          mountPath: /rootfs/proc
          readOnly: true
        - name: docker-socket
          mountPath: /var/run/docker.sock
        - name: utmp
          mountPath: /var/run/utmp
          readOnly: true
        - name: config
          mountPath: /etc/telegraf
      terminationGracePeriodSeconds: 30
      volumes:
      - name: sys
        hostPath:
          path: /sys
      - name: docker-socket
        hostPath:
          path: /var/run/docker.sock
      - name: proc
        hostPath:
          path: /proc
      - name: utmp
        hostPath:
          path: /var/run/utmp
      - name: config
        configMap:
          name: telegraf

This is the article that I followed to create a daemonset.

Here is the pods:

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
default       nginx-65f88748fd-jztrz               1/1     Running   0          7d18h
kube-system   coredns-fb8b8dccf-rl48l              1/1     Running   0          7d18h
kube-system   coredns-fb8b8dccf-x8fvx              1/1     Running   0          7d18h
kube-system   etcd-k8s-master                      1/1     Running   2          7d18h
kube-system   kube-apiserver-k8s-master            1/1     Running   2          7d18h
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          7d18h
kube-system   kube-flannel-ds-amd64-96tsl          1/1     Running   0          7d18h
kube-system   kube-flannel-ds-amd64-b884r          1/1     Running   0          7d18h
kube-system   kube-flannel-ds-amd64-pdqmq          1/1     Running   0          7d18h
kube-system   kube-proxy-42k2g                     1/1     Running   0          7d18h
kube-system   kube-proxy-77pw9                     1/1     Running   0          7d18h
kube-system   kube-proxy-n5mbs                     1/1     Running   0          7d18h
kube-system   kube-scheduler-k8s-master            1/1     Running   2          7d18h
monitoring    telegraf-dvtcl                       1/1     Running   5          117m
monitoring    telegraf-n2mqz                       1/1     Running   5          117m

tcpdump shows that something sent from the daemonset:

09:52:59.002901 IP 192.168.1.10.45546 > sdsfdsf.XmlIpcRegSvc: Flags [S], seq 3040818525, win 28200, options [mss 1410,sackOK,TS val 158999344 ecr 0,nop,wscale 7], length 0
E..<2.@.@......

y?...#..?5]......n(._.........
        z#0........................
09:52:59.002901 IP 192.168.1.10.45546 > sdsfdsf.XmlIpcRegSvc: Flags [S], seq 3040818525, win 28200, options [mss 1410,sackOK,TS val 158999344 ecr 0,nop,wscale 7], length 0
E..<2.@.@......

y?...#..?5]......n(._.........

But I can't see anything on our grafana dashboard. If I install a standalone rpm based telegraf on the nodes, it sents out and I can see the metrics. But I'm curious of the pod metrics.