We are using fargate type EKS pods and we would like to setup customized fluent bit logging, which will capture logs from different specified paths in different services containers and outputs them to cloudwatch.
For example,
service 1 keep logs at below paths:-
/var/log/nginx/*
/usr01/app/product/logs/*
service 2 keep logs at below paths:-
/jboss/product/logs/*
/birt/1.0.0/deploy/birt.war/logs"
/birt/1.0.0/logs/*
and so on ...
Therefore, we want to customize fluent bit configmap to capture logs generated in different paths for different services and they should be prefixed with the service name. Please help us to achieve that.
As explained in the documentation for Fargate logging, you cannot define input blocks for fluent-bit configuration.
https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html
In a typical Fluent Conf the main sections included are Service, Input, Filter, and Output. The Fargate log router however, only accepts:
The Filter and Output sections and manages the Service and Input sections itself.
The Parser section.
if you provide any other section than Filter, Output, and Parser, the sections are rejected.
You can try a sidecar container approach (which is a little more expensive) for your use case:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: example
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
volumeMounts:
- name: varlog
mountPath: /var/log/nginx
- name: fluent-bit
image: amazon/aws-for-fluent-bit:2.14.0
env:
- name: HOST_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
resources:
limits:
memory: 200Mi
requests:
cpu: 200m
memory: 100Mi
volumeMounts:
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
- name: varlog
mountPath: /var/log/nginx
readOnly: true
volumes:
- name: fluent-bit-config
configMap:
name: fluent-bit-config
- name: varlog
emptyDir: {}
terminationGracePeriodSeconds: 10
In this approach you use a fluentbit sidecar container and specify your custom settings for the input blocks. You can use this simple fluent-bit configmap as an example to test:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: example
labels:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
@INCLUDE application-log.conf
application-log.conf: |
[INPUT]
Name tail
Path /var/log/nginx/*.log
[OUTPUT]
Name stdout
Match *
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
You can test this approach by running the example above and sending an http request to the nginx pod. After that you should see in the fluent-bit sidecar logs something similar to this:
AWS for Fluent Bit Container Image Version 2.14.0
Fluent Bit v1.7.4
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2021/09/27 19:06:06] [ info] [engine] started (pid=1)
[2021/09/27 19:06:06] [ info] [storage] version=1.1.1, initializing...
[2021/09/27 19:06:06] [ info] [storage] in-memory
[2021/09/27 19:06:06] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/09/27 19:06:06] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/09/27 19:06:06] [ info] [sp] stream processor started
[2021/09/27 19:06:06] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1592225 watch_fd=1 name=/var/log/nginx/access.log
[2021/09/27 19:06:06] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1592223 watch_fd=2 name=/var/log/nginx/error.log
[0] tail.0: [1632769609.617430221, {"log"=>"10.42.0.139 - - [27/Sep/2021:19:06:49 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-""}]
[0] tail.0: [1632769612.241034110, {"log"=>"10.42.0.139 - - [27/Sep/2021:19:06:52 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-""}]