I'd like to send my logs to elastic search via fluent-bit. I've configured values.yaml
as follow.
parsers:
enabled: true
json:
- name: docker
timeKey: time
timeFormat: "%Y-%m-%dT%H:%M:%S.%L"
timeKeep: on
decodeFieldAs: json
backend:
type: es
es:
host: myhost
port: 9243
http_user: elastic
http_passwd: elastic
tls: "on"
Logs are coming in elastic search but log field is not decoded as JSON. Can you please help with this YAML to decode log
field as JSON.
Sample log/document generated by fluent-bit
{
"_index": "kubernetes_cluster-2019.03.30",
"_type": "flb_type",
"_id": "xdTVzGkBmTc6-uH5QzgK",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-03-30T04:09:02.259Z",
"log": "{\"time\":\"2019-03-30T04:09:02.258+00:00\",\"@version\":1,\"logger_name\":\"com.org.activemq.ActiveMQQueueUtility\",\"thread_name\":\"ORDER RESYNC TASK-0\",\"level\":\"INFO\",\"eventName\":\"syncOrder\",\"requestId\":\"LadyXhg0Hy8m7jJSQ2f\",\"eventMessage\":\"{\"sendEmail\":false,\"storeId\":61549}\",\"childRequestId\":\"LbBmKjCDuyaXQ-HwKL_\",\"action\":\"messagePublished\",\"isSent\":true,\"elapseTime\":101,\"queue\":\"HPT.SYNC.SYNC_O\",\"caller_class_name\":\"com.org.activemq.ActiveMQQueueUtility$ActiveMQProducer\",\"caller_method_name\":\"produce\",\"caller_file_name\":\"ActiveMQQueueUtility.java\",\"caller_line_number\":202}\n",
"stream": "stdout",
"time": "2019-03-30T04:09:02.259158471Z",
"kubernetes": {
"pod_name": "backend-c88bbb8f9-jtpfr",
"namespace_name": "dev",
"pod_id": "8700ba57-4d51-11e9-a90b-06fcff7cc9aa",
"labels": {
"app": "backend",
"pod-template-hash": "744666495",
"release": "dev"
},
"annotations": {
"checksum/config": "ceb71980bda81a95c3175a83f3d5cbe622c7e712d2c399a36d8045c8c4bcd467",
"checksum/secret": "eca5e141d20b020ec66cd82d784347e9550d01a139e494f9010ebd4e790538f1"
},
"host": "ip-xxx-xx-xx-xx.us-east-2.compute.internal",
"container_name": "backend",
"docker_id": "a2b8d61d0bd35e61f42a2524be8e1d04be96a2e7ce74b4620ce058cac2101357"
}
},
"fields": {
"@timestamp": [
"2019-03-30T04:09:02.259Z"
],
"time": [
"2019-03-30T04:09:02.259Z"
]
},
"sort": [
1553918942259
]
}