I am running one filebeat (version - 6.4.1) per node in kubernetes cluster with 1 master node and 3 worker nodes.

And a single logstash, elastic and Kibana for the entire cluster.

While the pods are up and running successfully, filebeat is unable to pull/send the logs to the logstash.

If I restart the filebeat pods then the logs can be seen from Kibana.

The error I see from filebeat logs are:

ERROR kubernetes/watcher.go:154 kubernetes: Watching API error EOF

Found a similar issue in ELK forums

https://discuss.elastic.co/t/kubernetes-filebeat-stops-sending-picking-up-logs/128578. It is said that filebeat of version 6.3.0 has a fix for this.

Component Versions:

cluster - 4 nodes (1 master & 3 workers)

master - 4 core & 8 GB RAM

worker - 16 core & 32 GB RAM

host OS - Centos: 7

container OS - alpine: 3.9.4

k8s - v1.13.1

docker - 18.09.0

filebeat - 6.4.1

logstash - 6.3.1

elasticsearch - 6.5.4

kibana - 6.5.4

I am facing the same issue in filebeat 6.4.1 > 6.3.0

Please suggest me if I need to make any changes in the ELK configurations.