I have 2 cluster
1. Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-eks-f17b81", GitCommit:"f17b810c9e5a82200d28b6210b458497ddfcf31b", GitTreeState:"clean", BuildDate:"2021-10-15T21:46:21Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6-gke.1500", GitCommit:"7ce0f9f1939dfc1aee910732e84cba03840df91e", GitTreeState:"clean", BuildDate:"2021-11-17T09:30:26Z", GoVersion:"go1.16.9b7", Compiler:"gc", Platform:"linux/amd64"}I use fluent-bit to tail contailer log files and push log to elasticsearch
In 1st k8s cluster, the container log 's format is:
{"log":"{\"method\":\"GET\",\"path\":\"/healthz\",\"format\":\"*/*\",\"controller\":\"Api::ApplicationController\",\"action\":\"healthz\",\"status\":204,\"duration\":0.61,\"view\":0.0,\"request_id\":\"4d54cc06-08d2-4487-b2d9-fabfb2286e89\",\"headers\":{\"SCRIPT_NAME\":\"\",\"QUERY_STRING\":\"\",\"SERVER_PROTOCOL\":\"HTTP/1.1\",\"SERVER_SOFTWARE\":\"puma 5.4.0 Super Flight\",\"GATEWAY_INTERFACE\":\"CGI/1.2\",\"REQUEST_METHOD\":\"GET\",\"REQUEST_PATH\":\"/healthz\",\"REQUEST_URI\":\"/healthz\",\"HTTP_VERSION\":\"HTTP/1.1\",\"HTTP_HOST\":\"192.168.95.192:80\",\"HTTP_USER_AGENT\":\"kube-probe/1.20+\",\"HTTP_ACCEPT\":\"*/*\",\"HTTP_CONNECTION\":\"close\",\"SERVER_NAME\":\"192.168.95.192\",\"SERVER_PORT\":\"80\",\"PATH_INFO\":\"/healthz\",\"REMOTE_ADDR\":\"192.168.79.131\",\"ROUTES_19640_SCRIPT_NAME\":\"\",\"ORIGINAL_FULLPATH\":\"/healthz\",\"ORIGINAL_SCRIPT_NAME\":\"\"},\"params\":{\"controller\":\"api/application\",\"action\":\"healthz\"},\"response\":{},\"custom\":{},\"@version\":\"dutycast-b2c-backend-v1.48.0-rc.5\",\"@timestamp\":\"2022-03-04T11:16:14.236Z\",\"message\":\"[204] GET /healthz (Api::ApplicationController#healthz)\"}\n","stream":"stdout","time":"2022-03-04T11:16:14.238067813Z"}It is in json format and I can parse easily using fluent-bit parser
And I do the same behavior for the 2nd k8s cluster but the the container log 's format is:
2022-03-04T11:19:24.050132912Z stdout F {"method":"GET","path":"/healthz","format":"*/*","controller":"Public::PublicPagesController","action":"healthz","status":204,"duration":0.52,"view":0.0,"request_id":"bcc799bb-5e5c-4758-9169-ecebb04b801f","headers":{"SCRIPT_NAME":"","QUERY_STRING":"","SERVER_PROTOCOL":"HTTP/1.1","SERVER_SOFTWARE":"puma 5.6.2 Birdie's Version","GATEWAY_INTERFACE":"CGI/1.2","REQUEST_METHOD":"GET","REQUEST_PATH":"/healthz","REQUEST_URI":"/healthz","HTTP_VERSION":"HTTP/1.1","HTTP_HOST":"10.24.0.22:3000","HTTP_USER_AGENT":"kube-probe/1.21","HTTP_ACCEPT":"*/*","HTTP_CONNECTION":"close","SERVER_NAME":"10.24.0.22","SERVER_PORT":"3000","PATH_INFO":"/healthz","REMOTE_ADDR":"10.24.0.1","ROUTES_71860_SCRIPT_NAME":"","ORIGINAL_FULLPATH":"/healthz","ORIGINAL_SCRIPT_NAME":"","ROUTES_71820_SCRIPT_NAME":""},"params":{"controller":"public/public_pages","action":"healthz"},"custom":null,"request_time":"2022-03-04T11:19:24.048+00:00","process_id":8,"@version":"vcam-backend-v0.1.0-rc24","response":"#\u003cActionDispatch::Response:0x00007f9d1f600888 @mon_data=#\u003cMonitor:0x00007f9d1f600838\u003e, @mon_data_owner_object_id=144760, @header={\"X-Frame-Options\"=\u003e\"ALLOW-FROM https://vietcapital.com.vn\", \"X-XSS-Protection\"=\u003e\"0\", \"X-Content-Type-Options\"=\u003e\"nosniff\", \"X-Download-Options\"=\u003e\"noopen\", \"X-Permitted-Cross-Domain-Policies\"=\u003e\"none\", \"Referrer-Policy\"=\u003e\"strict-origin-when-cross-origin\"}, @stream=#\u003cActionDispatch::Response::Buffer:0x00007f9d1f6045a0 @response=#\u003cActionDispatch::Response:0x00007f9d1f600888 ...\u003e, @buf=[\"\"], @closed=false, @str_body=nil\u003e, @status=204, @cv=#\u003cMonitorMixin::ConditionVariable:0x00007f9d1f600720 @monitor=#\u003cMonitor:0x00007f9d1f600838\u003e, @cond=#\u003cThread::ConditionVariable:0x00007f9d1f6006f8\u003e\u003e, @committed=false, @sending=false, @sent=false, @cache_control={}, @request=#\u003cActionDispatch::Request GET \"http://10.24.0.22:3000/healthz\" for 10.24.0.1\u003e\u003e","@timestamp":"2022-03-04T11:19:24.049Z","message":"[204] GET /healthz (Public::PublicPagesController#healthz)"}Both case we have same config log for the service, use the same fluent-bit version and same elasticsearch version, only different k8s cluster. In 2nd case, container log has been insert something like timestamp into begin of every entry log, and I can not parse this log 's format because it 's not json format.
I think kubernetes add default option into docker container 's log (https://docs.docker.com/engine/reference/commandline/logs/)
How can I fix format log into json format in 2nd case?