We are using fluent bit as a sidecar container on a Fargate EKS pod to push logs to cloud watch reading from a common volume mount from main application pod. It looks like permissions issue to us. There is no error in the logs though. Please check below.
ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: eretail
labels:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Log_Level info
Daemon off
Parsers_File parsers.conf
# HTTP_Server On
# HTTP_Listen 0.0.0.0
# HTTP_Port 2020
@INCLUDE application-log.conf
application-log.conf: |
[INPUT]
Name tail
Path /logs/boot/*.log
Tag boot.*
Parser docker
Mem_Buf_Limit 5MB
Refresh_Interval 10
[INPUT]
Name tail
Path /logs/access/*.log
Tag access.*
Parser docker
Mem_Buf_Limit 5MB
Refresh_Interval 10
[OUTPUT]
Name cloudwatch
Match *boot*
region ap-southeast-1
log_group_name eks-fluent-bit
log_stream_prefix ${HOSTNAME}-boot-log-
auto_create_group true
# workers 1
[OUTPUT]
Name cloudwatch
Match *access*
region ap-southeast-1
log_group_name eks-fluent-bit
log_stream_prefix ${HOSTNAME}-access-log-
auto_create_group true
# workers 1
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
Container Logs
AWS for Fluent Bit Container Image Version 2.14.0
Fluent Bit v1.7.4
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2021/10/04 10:36:01] [ info] [engine] started (pid=1)
[2021/10/04 10:36:01] [ info] [storage] version=1.1.1, initializing...
[2021/10/04 10:36:01] [ info] [storage] in-memory
[2021/10/04 10:36:01] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter log_group_name = 'eks-fluent-bit'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter default_log_group_name = 'fluentbit-default'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter log_stream_prefix = 'ip-17-225-20-45.ap-southeast-1.compute.internal-boot-log-'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter log_stream_name = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter default_log_stream_name = '/fluentbit-default'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter region = 'ap-southeast-1'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter log_key = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter role_arn = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter auto_create_group = 'true'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter new_log_group_tags = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter log_retention_days = '0'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter endpoint = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter sts_endpoint = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter credentials_endpoint = "
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 0] plugin parameter log_format = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter log_group_name = 'eks-fluent-bit'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter default_log_group_name = 'fluentbit-default'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter log_stream_prefix = 'ip-17-225-20-45.ap-southeast-1.compute.internal-access-log-'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter log_stream_name = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter default_log_stream_name = '/fluentbit-default'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter region = 'ap-southeast-1'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter log_key = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter role_arn = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter auto_create_group = 'true'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter new_log_group_tags = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter log_retention_days = '0'"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter endpoint = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter sts_endpoint = ''"
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter credentials_endpoint = "
time="2021-10-04T10:36:01Z" level=info msg="[cloudwatch 1] plugin parameter log_format = ''"
[2021/10/04 10:36:01] [ info] [sp] stream processor started
[2021/10/04 10:36:01] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1460720 watch_fd=1 name=/logs/boot/INTEGRATOR.log
[2021/10/04 10:36:01] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1460719 watch_fd=2 name=/logs/boot/server.logPolicy attached to the Fargate Pod Execution Role
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:DescribeLogStreams",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
Please help us to achieve this.