How to setup customized logging with Fluentbit in EKS Fargate?

9/24/2021

We are using fargate type EKS pods and we would like to setup customized fluent bit logging, which will capture logs from different specified paths in different services containers and outputs them to cloudwatch.

For example,

service 1 keep logs at below paths:-

/var/log/nginx/*
/usr01/app/product/logs/*

service 2 keep logs at below paths:-

/jboss/product/logs/*
/birt/1.0.0/deploy/birt.war/logs"
/birt/1.0.0/logs/*

and so on ...

Therefore, we want to customize fluent bit configmap to capture logs generated in different paths for different services and they should be prefixed with the service name. Please help us to achieve that.

-- Nitin Garg
amazon-eks
aws-fargate
fluent-bit
kubernetes
logging

1 Answer

9/25/2021

As explained in the documentation for Fargate logging, you cannot define input blocks for fluent-bit configuration.

https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html

In a typical Fluent Conf the main sections included are Service, Input, Filter, and Output. The Fargate log router however, only accepts:

The Filter and Output sections and manages the Service and Input sections itself.

The Parser section.

if you provide any other section than Filter, Output, and Parser, the sections are rejected.

You can try a sidecar container approach (which is a little more expensive) for your use case:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: example
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
          - containerPort: 80
        volumeMounts:
        - name: varlog
          mountPath: /var/log/nginx
      - name: fluent-bit
        image: amazon/aws-for-fluent-bit:2.14.0
        env:
          - name: HOST_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 200m
            memory: 100Mi
        volumeMounts:
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        - name: varlog
          mountPath: /var/log/nginx
          readOnly: true
      volumes:
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-config
        - name: varlog
          emptyDir: {}
      terminationGracePeriodSeconds: 10

In this approach you use a fluentbit sidecar container and specify your custom settings for the input blocks. You can use this simple fluent-bit configmap as an example to test:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: example
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush                     5
        Log_Level                 info
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Server               On
        HTTP_Listen               0.0.0.0
        HTTP_Port                 2020
        
    @INCLUDE application-log.conf
  
  application-log.conf: |
    [INPUT]
        Name                tail
        Path                /var/log/nginx/*.log
        
    [OUTPUT]
        Name                stdout
        Match               *
        
  parsers.conf: |
    [PARSER]
        Name                docker
        Format              json
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

You can test this approach by running the example above and sending an http request to the nginx pod. After that you should see in the fluent-bit sidecar logs something similar to this:

AWS for Fluent Bit Container Image Version 2.14.0
Fluent Bit v1.7.4
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/09/27 19:06:06] [ info] [engine] started (pid=1)
[2021/09/27 19:06:06] [ info] [storage] version=1.1.1, initializing...
[2021/09/27 19:06:06] [ info] [storage] in-memory
[2021/09/27 19:06:06] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/09/27 19:06:06] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/09/27 19:06:06] [ info] [sp] stream processor started
[2021/09/27 19:06:06] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1592225 watch_fd=1 name=/var/log/nginx/access.log
[2021/09/27 19:06:06] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1592223 watch_fd=2 name=/var/log/nginx/error.log
[0] tail.0: [1632769609.617430221, {"log"=>"10.42.0.139 - - [27/Sep/2021:19:06:49 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-""}]
[0] tail.0: [1632769612.241034110, {"log"=>"10.42.0.139 - - [27/Sep/2021:19:06:52 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-""}]
-- Emidio Neto
Source: StackOverflow