fluent-bit cannot parse kubernetes logs

7/19/2020

I would like to forward Kubernetes logs from fluent-bit to elasticsearch through fluentd but fluent-bit cannot parse kubernetes logs properly. In order to install Fluent-bit and Fluentd, I use Helm charts. I tried both stable/fluentbit and fluent/fluentbit and faced with same problem:

#0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'Could not dynamically add mapping for field [app.kubernetes.io/component]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text].'"

I put following lines into fluent-bit values file as shown here

  remapMetadataKeysFilter:
    enabled: true
    match: kube.*

    ## List of the respective patterns and replacements for metadata keys replacements
    ## Pattern must satisfy the Lua spec (see https://www.lua.org/pil/20.2.html)
    ## Replacement is a plain symbol to replace with
    replaceMap:
      - pattern: "[/.]"
        replacement: "_"

...nothing changed, same errors are listed.

Is there a workaround to get rid of that bug?

my values.yaml is here:

    # Default values for fluent-bit.

# kind -- DaemonSet or Deployment
kind: DaemonSet

# replicaCount -- Only applicable if kind=Deployment
replicaCount: 1

image:
  repository: fluent/fluent-bit
  pullPolicy: Always
  # tag:

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  create: true
  annotations: {}
  name:

rbac:
  create: true

podSecurityPolicy:
  create: false

podSecurityContext:
  {}
  # fsGroup: 2000

securityContext:
  {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

service:
  type: ClusterIP
  port: 2020
  annotations:
    prometheus.io/path: "/api/v1/metrics/prometheus"
    prometheus.io/port: "2020"
    prometheus.io/scrape: "true"

serviceMonitor:
  enabled: true
  namespace: monitoring
  interval: 10s
  scrapeTimeout: 10s
  # selector:
  #  prometheus: my-prometheus

resources:
  {}
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

podAnnotations: {}

priorityClassName: ""

env: []

envFrom: []

extraPorts: []
#   - port: 5170
#     containerPort: 5170
#     protocol: TCP
#     name: tcp

extraVolumes: []

extraVolumeMounts: []

## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit
config:
  ## https://docs.fluentbit.io/manual/service
  service: |
    [SERVICE]
        Flush 1
        Daemon Off
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser docker
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

    [FILTER]
        Name    lua
        Match   kube.*
        script  /fluent-bit/etc/functions.lua
        call    dedot
        
  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name          forward
        Match         *
        Host          fluentd-in-forward.elastic-system.svc.cluster.local
        Port          24224
        tls           off
        tls.verify    off

  ## https://docs.fluentbit.io/manual/pipeline/parsers
  customParsers: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
-- Tireli Efe
elasticsearch
fluent-bit
fluentd
helmfile
kubernetes

2 Answers

7/20/2020

I think that your problem isn't in kubernetes, isn't in fluentbit/fluentd chart, your problem is in elasticsearch, concretely in the mapping.

In elsticsearch version 7.x you can't have different types (string, int, etc) for the same field.

For solve thus issue, i use "ignore_malformed": true in the index template that i use for the kubernetes logs.

https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html

The malformed field is not indexed, but other fields in the document are processed normally.

-- Víctor Oriol
Source: StackOverflow

1/8/2021

I had the same issue, which is caused by multiple labels that are converted into json. I renamed the conflicting keys to match with the newer format of recommended labels:

<filter **>
  @type rename_key
  rename_rule1 ^app$ app.kubernetes.io/name
  rename_rule2 ^chart$ helm.sh/chart
  rename_rule3 ^version$ app.kubernetes.io/version
  rename_rule4 ^component$ app.kubernetes.io/component
  rename_rule5 ^istio$ istio.io/name
</filter>
-- Jacob Stampe Mikkelsen
Source: StackOverflow