Unable to connect Fluend to ElasticSearch

11/5/2020

I am following instructions to connect FluentD to Elastic from HERE.

Unfortunately, when I deploy the daemonset, I get the following error:

2020-11-05 09:13:19 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. no implicit conversion of nil into String (TypeError)
2020-11-05 09:13:19 +0000 [warn]: #0 [out_es] Remaining retry: 14. Retry to communicate after 2 second(s).
2020-11-05 09:13:23 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. no implicit conversion of nil into String (TypeError)
2020-11-05 09:13:23 +0000 [warn]: #0 [out_es] Remaining retry: 13. Retry to communicate after 4 second(s).
2020-11-05 09:13:31 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. no implicit conversion of nil into String (TypeError)
2020-11-05 09:13:31 +0000 [warn]: #0 [out_es] Remaining retry: 12. Retry to communicate after 8 second(s).
2020-11-05 09:13:47 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. no implicit conversion of nil into String (TypeError)
2020-11-05 09:13:47 +0000 [warn]: #0 [out_es] Remaining retry: 11. Retry to communicate after 16 second(s).
2020-11-05 09:14:24 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. no implicit conversion of nil into String (TypeError)
2020-11-05 09:14:24 +0000 [warn]: #0 [out_es] Remaining retry: 10. Retry to communicate after 32 second(s).
2020-11-05 09:15:28 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. no implicit conversion of nil into String (TypeError)
2020-11-05 09:15:28 +0000 [warn]: #0 [out_es] Remaining retry: 9. Retry to communicate after 64 second(s).
#<Thread:0x00007fcf20f41a40@/fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265 run> terminated with exception (report_on_exception is true):
/usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in `sysread': error reading from socket: Connection reset by peer (HTTP::ConnectionError)
from /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in `readpartial'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in `readpartial'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:212:in `read_more'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:92:in `readpartial'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:30:in `readpartial'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:36:in `each'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.1/lib/kubeclient/watch_stream.rb:25:in `each'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in `start_namespace_watch'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265:in `block in configure'
/usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in `sysread': Connection reset by peer (Errno::ECONNRESET)
from /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in `readpartial'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in `readpartial'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:212:in `read_more'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:92:in `readpartial'
from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:30:in `readpartial'

I have tried both true and false for this flag:

  - name: FLUENT_ELASTICSEARCH_SSL_VERIFY
            value: "false"

and have had no luck.

I would appreciate any pointers as to what might be the issue.

-- Samuel Dare
azure-aks
elasticsearch
fluentd
kubernetes

1 Answer

11/5/2020

You are passing one of the env variables a blank value. Update that.

-- avadhut007
Source: StackOverflow