I hope someone can help me with setup comminication with ELK on Kubernetes cluster and some suggestion on how to do that for a following task:

Jenkins on a separate machine after execution of a job has a log file (not a Jenkins logs but some logs produced by a code) that I need to send via network to ELK in order to process it.

First of all I'm a beginner with Kubernetes and Elastic Stack I've setup Jenkins, ElasticStack (Fluentd, ElasticSearch, Kibana and they communicate with each other, tested on kubernetes logs)

So I have numerous questions by now)

1. In the example I was following Fluentd is setup as a DaemonSet - means a fluentd pod is going to run on every node. I guess it is suggested for processing local log file which is not an option for me. I think I need to deploy a regular Fluentd pod and start a service that with do port forwarding to Fluentd port on a node, is that correct?

2. I set test config for Fluentd to accept TCP connection and print results:

  <source>
    @type tcp
    tag "tcp.events"
    port 5000
    bind "0.0.0.0"
    delimiter "\\n"
    <parse>
      @type "regexp"
      expression "/^(?<field>\\w+)$/"
    </parse>
  </source>
  <match **>
    @type stdout
  </match>

Since I can't connect from outside I was trying to send tcp package from cluster node with kubernetes:

 kubectl exec fluentd-l5r46 echo '123456:awesome' | netcat 0.0.0.0 5000

and observe results in logs:

kubectl logs pod/fluentd-l5r46

But no new messages appear there, only initialization one:

2020-02-25 10:52:55 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-02-25 10:52:55 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tcp
    tag "tcp.events"
    port 5000
    bind "0.0.0.0"
    delimiter "\\n"
    <parse>
      @type "regexp"
      expression "/^(?<field>\\w+)$/"
    </parse>
  </source>
  <match **>
    @type stdout
  </match>
</ROOT>
2020-02-25 10:52:55 +0000 [info]: starting fluentd-1.1.3 pid=7 ruby="2.3.3"
2020-02-25 10:52:55 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby2.3", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
2020-02-25 10:52:55 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.9.2'
2020-02-25 10:52:55 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.0.0'
2020-02-25 10:52:55 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.0'
2020-02-25 10:52:55 +0000 [info]: gem 'fluentd' version '1.1.3'
2020-02-25 10:52:55 +0000 [info]: adding match pattern="**" type="stdout"
2020-02-25 10:52:55 +0000 [info]: adding source type="tcp"
2020-02-25 10:52:55 +0000 [info]: #0 starting fluentd worker pid=11 ppid=7 worker=0
2020-02-25 10:52:55 +0000 [info]: #0 fluentd worker is now running worker=0
2020-02-25 10:52:55.875310545 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}

So question is any suggestion regarding an solution architecture for this task, because I think I do not completely understand how to properly setup Fluentd for this and any suggestions regarding tcp connection

Thanks!