Fluent bit - TCP Output plugin - Constant connection and JSON stream

1/16/2020

I have been trying to configure Fluent bit to TCP output plugin. As far as I could observe Fluent bit has been initiating and closing a TCP connection to destination server in every JSON log flush. However, we would like to have a constant connection and stream logs over it.

Closest configured option is "Format json_stream" for TCP output plugin. Is supposed to Fluentbit establish and keep a connection to stream over TCP or every flush requires a new connection?

It is a daemonset consisted of a Fluentbit container communicating through localhost to server container.

Very grateful for any help on that!

 "filter-kubernetes.conf" = <<-EOT
    [FILTER]
        Name                kubernetes
        Match               app.*
        Kube_URL            https://kubernetes.default.svc:443
        Merge_Log           On
        Merge_Log_Trim      On
        Keep_Log            Off
        K8S-Logging.Parser  Off
    [FILTER]
        Name                grep
        Match               app.*
        Regex               stream stdout
    [FILTER]
        Name                record_modifier
        Match               app.*
        Remove_key          annotations
EOT
"fluent-bit.conf"        = <<-EOT
    [SERVICE]
        Flush               1
        Log_Level           info
        Daemon              off
        Parsers_File        parsers.conf
        HTTP_Server         On
        HTTP_Listen         0.0.0.0
        HTTP_Port           2020
    @INCLUDE input-sources.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-nats.conf
EOT
"input-sources.conf"     = <<-EOT
    [INPUT]
        Name                tail
        Tag                 app.*
        Path                /var/log/containers/*.log
        Parser              docker
        DB                  /var/log/flb_kube.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
EOT
    [OUTPUT]
        Name                tcp
        Match               *car*
        Format              json_stream
        json_date_format    iso8601
        json_date_key       timestamp
        Host                127.0.0.1
        Port                1111
EOT
"parsers.conf"           = <<-EOT
    [PARSER]
        Name                docker
        Format              json
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%L
        Time_Keep           On
EOT
  }
}
-- rsilva
fluent-bit
fluentd
kubernetes
logging
tcp

0 Answers