Configure fluent-bit helm chart for docker

3/30/2019

I'd like to send my logs to elastic search via fluent-bit. I've configured values.yaml as follow.

parsers:
  enabled: true
  json:
    - name: docker
      timeKey: time
      timeFormat: "%Y-%m-%dT%H:%M:%S.%L"
      timeKeep: on
      decodeFieldAs: json
backend:
  type: es
  es:
    host: myhost
    port: 9243
    http_user: elastic
    http_passwd: elastic
    tls: "on"

Logs are coming in elastic search but log field is not decoded as JSON. Can you please help with this YAML to decode log field as JSON.

Sample log/document generated by fluent-bit

{
  "_index": "kubernetes_cluster-2019.03.30",
  "_type": "flb_type",
  "_id": "xdTVzGkBmTc6-uH5QzgK",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-03-30T04:09:02.259Z",
    "log": "{\"time\":\"2019-03-30T04:09:02.258+00:00\",\"@version\":1,\"logger_name\":\"com.org.activemq.ActiveMQQueueUtility\",\"thread_name\":\"ORDER RESYNC TASK-0\",\"level\":\"INFO\",\"eventName\":\"syncOrder\",\"requestId\":\"LadyXhg0Hy8m7jJSQ2f\",\"eventMessage\":\"{\"sendEmail\":false,\"storeId\":61549}\",\"childRequestId\":\"LbBmKjCDuyaXQ-HwKL_\",\"action\":\"messagePublished\",\"isSent\":true,\"elapseTime\":101,\"queue\":\"HPT.SYNC.SYNC_O\",\"caller_class_name\":\"com.org.activemq.ActiveMQQueueUtility$ActiveMQProducer\",\"caller_method_name\":\"produce\",\"caller_file_name\":\"ActiveMQQueueUtility.java\",\"caller_line_number\":202}\n",
    "stream": "stdout",
    "time": "2019-03-30T04:09:02.259158471Z",
    "kubernetes": {
      "pod_name": "backend-c88bbb8f9-jtpfr",
      "namespace_name": "dev",
      "pod_id": "8700ba57-4d51-11e9-a90b-06fcff7cc9aa",
      "labels": {
        "app": "backend",
        "pod-template-hash": "744666495",
        "release": "dev"
      },
      "annotations": {
        "checksum/config": "ceb71980bda81a95c3175a83f3d5cbe622c7e712d2c399a36d8045c8c4bcd467",
        "checksum/secret": "eca5e141d20b020ec66cd82d784347e9550d01a139e494f9010ebd4e790538f1"
      },
      "host": "ip-xxx-xx-xx-xx.us-east-2.compute.internal",
      "container_name": "backend",
      "docker_id": "a2b8d61d0bd35e61f42a2524be8e1d04be96a2e7ce74b4620ce058cac2101357"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-03-30T04:09:02.259Z"
    ],
    "time": [
      "2019-03-30T04:09:02.259Z"
    ]
  },
  "sort": [
    1553918942259
  ]
}
-- Vicky Thakor
docker
fluent-bit
kubernetes

0 Answers