I run filebeat as a kubernetes daemonset in kubernetes cluster with autodiscover turned on and I have a errors in filebeat log:
"ERROR kubernetes/util.go:90 kubernetes: Querying for pod failed with error: performing request: Get https://10.96.0.1:443/api/v1/namespaces/default/pods/filebeat-daemon-7nkqt: dial tcp 10.96.0.1:443: connect: connection refused
ERROR kubernetes/watcher.go:183 kubernetes: Performing a resource sync err performing request: Get https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0:
dial tcp 10.96.0.1:443: connect: connection refused for *v1.PodList".
When I add a command "sleep 5" into filebeat start command line, errors disappear. Does anyone have a solution for this problem?
Here is a filebeat manifest:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config-json
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.registry_file: "/data/logs/elog/filebeat/registry"
filebeat.autodiscover:
providers:
- type: kubernetes
include_annotations: ["logging"]
templates:
- condition:
equals:
kubernetes.annotations.logging: "json"
config:
- type: docker
json:
keys_under_root: false
add_error_key: false
ignore_decoding_error: true
message_key: message
overwrite_keys: true
containers.ids:
- "${data.kubernetes.container.id}"
processors:
- drop_event: # drop others if any
when:
not.equals:
kubernetes.annotations.logging: "json"
- add_cloud_metadata: ~
- add_kubernetes_metadata:
in_cluster: true
processors:
- drop_fields:
fields: ["host"]
output.logstash:
hosts: ["logstash-host:5044"]
logging:
to_files: true
files:
path: "/data/logs/elog/filebeat"
name: filebeat.log
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
level: info
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat-daemon
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
name: filebeat
k8s-app: filebeat
template:
metadata:
labels:
name: filebeat
k8s-app: filebeat
spec:
containers:
- name: filebeat-container
image: docker-registry.company.com/elog/filebeat-test2
command: ['/bin/bash', '-c', 'ln -f -s /data/configs/filebeat/filebeat.yml /data/elog/filebeat/current/filebeat.yml;sleep 5;/startup_filebeat.sh']
volumeMounts:
- mountPath: /var/lib/docker/containers
name: filebeat-log-source
- name: filebeat-config
mountPath: /data/configs/filebeat
- name: varlog
mountPath: /var/log
readOnly: true
imagePullSecrets:
- name: regcred
volumes:
- name: filebeat-log-source
hostPath:
path: /var/lib/docker/containers
type: DirectoryOrCreate
- name: filebeat-config
configMap:
defaultMode: 0644
name: filebeat-config-json
- name: varlog
hostPath:
path: /var/log
type: DirectoryOrCreate