Add additional field in fluentd

12/27/2019

I have a messesae as below

{"log":"kubernetes.var.log.dev-2019-12-24.log\u0009{\"msg\":\"[2019-12-24 10:34:58] app.ERROR: [ApiExceptionHandler:onKernelException]: default not match= exception is Symfony\\\\Component\\\\HttpKernel\\\\Exception\\\\NotFoundHttpException [] []\"}\n","stream":"stdout","time":"2019-12-24T10:34:58.295814385Z"}

Now I want to split it in 4 parts:

file_name: kubernetes.var.log.dev-2019-12-24.log
time: 2019-12-24 10:34:58
messeage_type: app.ERROR
msg: all remainding messeage

In fluentd configuration, I set a regex like:

  <parse>
    @type "regexp"
    expression [(?<time>.+)] (?<kind>.*ERROR|.*INFO): (?<msg>.*)$
  </parse>

but it not work as expect, it show a warning

2019-12-27 02:34:30 +0000 [warn]: [fluentd-containers.log] pattern not match: "{\"log\":\"kubernetes.var.log.dev-2019-12-27.log\\u0009{\\\"msg\\\":\\\"[2019-12-27 02:34:27] security.INFO: Populated the TokenStorage with an anonymous Token. [] []\\\"}\\n\",\"stream\":\"stdout\",\"time\":\"2019-12-27T02:34:30.699454425Z\"}"
2019-12-27 02:34:30 +0000 [warn]: [fluentd-containers.log] pattern not match: "{\"log\":\"kubernetes.var.log.dev-2019-12-27.log\\u0009{\\\"msg\\\":\\\"[2019-12-27 02:34:27] app.INFO: [UserCtrl:Login]: request_data= {\\\\\\\"email\\\\\\\":\\\\\\\"tui@gmail.com\\\\\\\",\\\\\\\"password\\\\\\\":\\\\\\\"asfasfd\\\\\\\"} [] []\\\"}\\n\",\"stream\":\"stdout\",\"time\":\"2019-12-27T02:34:30.699458964Z\"}"

I think there is something wrong in regex, please advise me how to fix it

-- Tien Dung Tran
fluentd
kubernetes
regex

1 Answer

12/27/2019

You need to escape [ and ]:

expression \[(?<time>.+)\] (?<kind>.*ERROR|.*INFO): (?<msg>.*)$
-- Anonymous
Source: StackOverflow