I have Elasticsearch running on Kubernetes (EKS), with filebeat running as daemonset on Kubernetes.

Now I am trying to get the logs from other EC2 machines (outside of the EKS), so have installed exact version of filebeat on EC2 and configured it to send logs to Elasticsearch running on Kubernetes.

But not able to see any logs in Elasticsearch (Kibana). Here are the logs for filebeat

2019-08-26T18:18:16.005Z    INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.2.1
2019-08-26T18:18:16.005Z    INFO    [index-management]  idxmgmt/std.go:178  Set output.elasticsearch.index to 'filebeat-7.2.1' as ILM is enabled.
2019-08-26T18:18:16.005Z    INFO    elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200
2019-08-26T18:18:16.005Z    INFO    add_cloud_metadata/add_cloud_metadata.go:351    add_cloud_metadata: hosting provider type detected as aws, metadata={"availability_zone":"us-west-2a","instance":{"id":"i-0185e1d68306f95b4"},"machine":{"type":"t2.medium"},"provider":"aws","region":"us-west-2"}
2019-08-26T18:18:16.005Z    INFO    [publisher] pipeline/module.go:97   Beat name: dev-web1
2019-08-26T18:18:16.006Z    INFO    elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200

Not much info in the logs.

Then I notice :

root@dev-web1:~# sudo systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-08-26 18:18:47 UTC; 18min ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 7768 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─7768 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs

Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(http://elasticsear
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(elasticsearch(http://elastic
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        INFO        [publisher]        pipeline/retry.go:189        retryer: send unwait-signal to consumer
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:191          done
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:166        retryer: send wait signal to consumer
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:168          done
Aug 26 18:35:47 dev-web1 filebeat[7768]: 2019-08-26T18:35:47.028Z        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitori
Aug 26 18:36:17 dev-web1 filebeat[7768]: 2019-08-26T18:36:17.028Z        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitori
root@dev-web1:~#

But I can't read complete line in above status message.

So I tried :

root@dev-web1:~# curl elasticsearch.dev.domain.net/_cat/health
1566844775 18:39:35 dev-eks-logs green 3 3 48 24 0 0 0 0 - 100.0%
root@dev-web1:~#

which worked but not with port

root@dev-web1:~# curl elasticsearch.dev.domain.net:9200/_cat/health

filebeat has following config

output.elasticsearch:
  hosts: ["elasticsearch.dev.domain.net"]
  username: "elastic"
  password: "changeme"

How can I fix this at filebeat side ?

Telnet Test :

root@dev-web1:~# telnet <ip> 5044
Trying <ip>...
telnet: Unable to connect to remote host: Connection refused
root@dev-web1:~# telnet localhost 5044
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
root@dev-web1:~#