I have been trying to configure Fluent bit to TCP output plugin. As far as I could observe Fluent bit has been initiating and closing a TCP connection to destination server in every JSON log flush. However, we would like to have a constant connection and stream logs over it.
Closest configured option is "Format json_stream" for TCP output plugin. Is supposed to Fluentbit establish and keep a connection to stream over TCP or every flush requires a new connection?
It is a daemonset consisted of a Fluentbit container communicating through localhost to server container.
Very grateful for any help on that!
"filter-kubernetes.conf" = <<-EOT
[FILTER]
Name kubernetes
Match app.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
Merge_Log_Trim On
Keep_Log Off
K8S-Logging.Parser Off
[FILTER]
Name grep
Match app.*
Regex stream stdout
[FILTER]
Name record_modifier
Match app.*
Remove_key annotations
EOT
"fluent-bit.conf" = <<-EOT
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
@INCLUDE input-sources.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE output-nats.conf
EOT
"input-sources.conf" = <<-EOT
[INPUT]
Name tail
Tag app.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
EOT
[OUTPUT]
Name tcp
Match *car*
Format json_stream
json_date_format iso8601
json_date_key timestamp
Host 127.0.0.1
Port 1111
EOT
"parsers.conf" = <<-EOT
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
EOT
}
}