How to connect filebeat container to logstash kubernetes pod running on different machine?

1/20/2022

We are running filebeat in one machine as container.

Docker File:

version: '3'
services:
  filebeat:
    container_name: filebeat
    image: docker.elastic.co/beats/filebeat:7.5.0
    restart: always
    user: root
    volumes:
      - filebeat:/usr/share/filebeat/data
      - ./mylog:/usr/share/filebeat/mylog
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
    command: ["--strict.perms=false"]
    ulimits:
      memlock:
        soft: -1
        hard: -1
    stdin_open: true
    tty: true
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "50"
volumes:
  filebeat:

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/mylog/*.log

output.logstash:
  hosts: ["IP:31737"]

And a logstash as kubernetes pod in another machine

logstash-input.yml

input {
  beats {
    host => "0.0.0.0"
    port => 31737
  }
}

IP:31737 => node ip and port

But still filebeat is not able to connect to logstash and gives below error.

2022-01-20T11:46:26.751Z        ERROR   logstash/async.go:256   Failed to publish events caused by: lumberjack protocol error
2022-01-20T11:46:26.751Z        ERROR   logstash/async.go:256   Failed to publish events caused by: lumberjack protocol error
2022-01-20T11:46:26.751Z        INFO    [publisher]     pipeline/retry.go:173   retryer: send wait signal to consumer
2022-01-20T11:46:26.752Z        INFO    [publisher]     pipeline/retry.go:175     done
2022-01-20T11:46:26.756Z        ERROR   logstash/async.go:256   Failed to publish events caused by: client is not connected
2022-01-20T11:46:28.235Z        ERROR   pipeline/output.go:121  Failed to publish events: client is not connected
2022-01-20T11:46:28.235Z        INFO    pipeline/output.go:95   Connecting to backoff(async(tcp://IP:31737))
2022-01-20T11:46:28.235Z        INFO    pipeline/output.go:105  Connection to backoff(async(tcp://IP:31737)) established
2022-01-20T11:46:28.269Z        INFO    [publisher]     pipeline/retry.go:196   retryer: send unwait-signal to consumer
2022-01-20T11:46:28.269Z        INFO    [publisher]     pipeline/retry.go:198     done
2022-01-20T11:46:28.270Z        ERROR   logstash/async.go:256   Failed to publish events caused by: lumberjack protocol error
2022-01-20T11:46:28.270Z        ERROR   logstash/async.go:256   Failed to publish events caused by: lumberjack protocol error
2022-01-20T11:46:28.270Z        INFO    [publisher]     pipeline/retry.go:173   retryer: send wait signal to consumer
2022-01-20T11:46:28.270Z        INFO    [publisher]     pipeline/retry.go:175     done
2022-01-20T11:46:28.274Z        ERROR   logstash/async.go:256   Failed to publish events caused by: client is not connected
2022-01-20T11:46:30.253Z        ERROR   pipeline/output.go:121  Failed to publish events: client is not connected
2022-01-20T11:46:30.253Z        INFO    pipeline/output.go:95   Connecting to backoff(async(tcp://IP:31737))
2022-01-20T11:46:30.253Z        INFO    pipeline/output.go:105  Connection to backoff(async(tcp://IP:31737)) established
2022-01-20T11:46:30.288Z        INFO    [publisher]     pipeline/retry.go:196   retryer: send unwait-signal to consumer
2022-01-20T11:46:30.288Z        INFO    [publisher]     pipeline/retry.go:198     done
2022-01-20T11:46:30.288Z        ERROR   logstash/async.go:256   Failed to publish events caused by: lumberjack protocol error
2022-01-20T11:46:30.288Z        ERROR   logstash/async.go:256   Failed to publish events caused by: lumberjack protocol error
2022-01-20T11:46:30.288Z        INFO    [publisher]     pipeline/retry.go:173   retryer: send wait signal to consumer
2022-01-20T11:46:30.288Z        INFO    [publisher]     pipeline/retry.go:175     done
2022-01-20T11:46:30.293Z        ERROR   logstash/async.go:256   Failed to publish events caused by: client is not connected

Any help appreciated.

Update: Telnet is working from both machine as well as from docker.

-- Dhyanesh Naik
filebeat
kubernetes
logstash
logstash-configuration
logstash-file

0 Answers