fluentd is not able to connect aws elasticsearch from eks

10/7/2020

I have installed a aws-es-proxy helm chart. I am using https://github.com/kokuwaio/helm-charts/tree/main/charts/fluentd-elasticsearch repo for created fluentd. 2020-10-07 04:34:38 +0000 error: unexpected error error_class=NoMethodError error="undefined method []' for nil:NilClass" 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.1.0/lib/fluent/plugin/out_elasticsearch.rb:464:indetect_es_major_version' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.1.0/lib/fluent/plugin/out_elasticsearch.rb:455:in block in handle_last_seen_es_major_version' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.1.0/lib/fluent/plugin/elasticsearch_index_template.rb:35:inretry_operate' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.1.0/lib/fluent/plugin/out_elasticsearch.rb:454:in handle_last_seen_es_major_version' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.1.0/lib/fluent/plugin/out_elasticsearch.rb:312:inconfigure' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/plugin.rb:173:in configure' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/agent.rb:132:inadd_match' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/agent.rb:74:in block in configure' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/agent.rb:64:ineach' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/agent.rb:64:in configure' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/label.rb:31:inconfigure' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/root_agent.rb:143:in block in configure' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/root_agent.rb:143:ineach' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/root_agent.rb:143:in configure' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/engine.rb:105:inconfigure' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/engine.rb:80:in run_configure' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/supervisor.rb:602:inblock in run_worker' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/supervisor.rb:840:in main_process' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/supervisor.rb:594:inrun_worker' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/gems/fluentd-1.11.1/lib/fluent/command/fluentd.rb:361:in <top (required)>' 2020-10-07 04:34:38 +0000 [error]: /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:inrequire' 2020-10-07 04:34:38 +0000 error: /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in require' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/gems/fluentd-1.11.1/bin/fluentd:8:in<top (required)>' 2020-10-07 04:34:38 +0000 error: /usr/local/bundle/bin/fluentd:23:in load' 2020-10-07 04:34:38 +0000 [error]: /usr/local/bundle/bin/fluentd:23:in<main>'enter code here

-- Dinesh
amazon-eks
aws-elasticsearch
elasticsearch
fluentd
kubernetes

1 Answer

10/11/2020

I advise you to do this workshop in particular this step: https://www.eksworkshop.com/intermediate/230_logging/ (EKS-logging fluetbit + elasticsearch + kibana)

I just tried this and found no problems.

-- andrea catalano
Source: StackOverflow