K
Q

Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached

12/28/2016

I am trying to set up EFK (elasticsearch, fluentd, kibana) on kubernetes cluster, so i used the following controller and service yaml files:

fluentd-es.yaml

https://github.com/kubernetes/kubernetes/blob/release-1.2/cluster/saltbase/salt/fluentd-es/fluentd-es.yaml

es-controller.yaml, es-service.yaml, kibana-controller.yaml and kibana-service.yaml

https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

after running them, i had the following log output and kibana dashboard was unable to show me logs and charts (keep loading for ever like next image).

enter image description here

fluentd log snapshot:

enter image description here enter image description here

elasticsearch log snapshot:

enter image description here

kibana log snapshot

enter image description here

-- mootez
elasticsearch
fluentd
kibana
kubernetes

Similar Questions

Kubernetes failed to setup network for pod after executed kubeadm reset
Deploying a network of containers in a single K8s pod
AWS instance can't be accessed from browser
How to identify schedulable nodes in Kubernetes

2 Answers

8/21/2017

You have two issues:

  1. ES connection is gettng time out after retrying.So make sure you are defining right es config in fluentd.conf.

  2. Also this is giving- BufferQueueLimitError which comes when your queue is filled due to connection time out. If you are expecting to fix this you should define:

    buffered memory and buffer_type memory buffer_chunk_limit **m buffer_queue_limit ** flush_interval ***s disable_retry_limit false retry_wait **s

refer- https://docs.fluentd.org/v0.12/articles/buffer-plugin-overview#secondary-output

-- ankush singh
Source: StackOverflow
1/3/2017

The logs are pretty much telling you .... there's a connection problem to Elasticsearch.

-- tyrell_c
Source: StackOverflow