How to fix "connection refused" error during filebeat start in kubernetes

7/10/2019

Error starting filebeat as daemonset in kubernetes

I run filebeat as a kubernetes daemonset in kubernetes cluster with autodiscover turned on and I have a errors in filebeat log:

"ERROR   kubernetes/util.go:90   kubernetes: Querying for pod failed with error: performing request: Get https://10.96.0.1:443/api/v1/namespaces/default/pods/filebeat-daemon-7nkqt: dial tcp 10.96.0.1:443: connect: connection refused
ERROR   kubernetes/watcher.go:183       kubernetes: Performing a resource sync err performing request: Get https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0:
dial tcp 10.96.0.1:443: connect: connection refused for *v1.PodList".

When I add a command "sleep 5" into filebeat start command line, errors disappear. Does anyone have a solution for this problem?

Here is a filebeat manifest:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config-json
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.registry_file: "/data/logs/elog/filebeat/registry"
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          include_annotations: ["logging"]
          templates:
            - condition:
                equals:
                  kubernetes.annotations.logging: "json"
              config:
                - type: docker
                  json:
                    keys_under_root: false
                    add_error_key: false
                    ignore_decoding_error: true
                    message_key: message
                    overwrite_keys: true  
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  processors:
                    - drop_event: # drop others if any
                        when:
                            not.equals:
                              kubernetes.annotations.logging: "json"
                    - add_cloud_metadata: ~
                    - add_kubernetes_metadata:
                        in_cluster: true
    processors:
      - drop_fields:
          fields: ["host"]
    output.logstash:
      hosts: ["logstash-host:5044"]
    logging:
      to_files: true
      files:
        path: "/data/logs/elog/filebeat"
        name: filebeat.log
        rotateeverybytes: 10485760 # = 10MB
        keepfiles: 7
      level: info

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat-daemon
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      name: filebeat
      k8s-app: filebeat
  template:
    metadata:
      labels:
        name: filebeat
        k8s-app: filebeat
    spec:
      containers:
        - name: filebeat-container
          image: docker-registry.company.com/elog/filebeat-test2
          command: ['/bin/bash', '-c', 'ln -f -s /data/configs/filebeat/filebeat.yml /data/elog/filebeat/current/filebeat.yml;sleep 5;/startup_filebeat.sh']
          volumeMounts:
            - mountPath: /var/lib/docker/containers
              name: filebeat-log-source
            - name: filebeat-config
              mountPath: /data/configs/filebeat
            - name: varlog
              mountPath: /var/log
              readOnly: true
      imagePullSecrets:
        - name: regcred
      volumes:
        - name: filebeat-log-source
          hostPath:
            path: /var/lib/docker/containers
            type: DirectoryOrCreate
        - name: filebeat-config
          configMap:
            defaultMode: 0644
            name: filebeat-config-json
        - name: varlog
          hostPath:
            path: /var/log
            type: DirectoryOrCreate
-- maksim
filebeat
kubernetes

0 Answers