K
Q

How to change the severity level (INFO, ERROR, WARNING, etc.) of log message in Fluentd based on some keyword in the log payload?

10/17/2017

I really hope somebody would help me out with this question, as I have been trying to figure it out for days.

I have container running in kubernetes in GKE. In /var/log/containers/my_container.log, I have something like this (among some other logs with different formats):

{"log":"17-Oct-2017;04:36:29.744 : [main] [server:] [id:] [yt:] ERROR no.myproject.service.Server - call failed for some reason\n","stream":"stdout","time":"2017-10-17T04:36:29.750702216Z"}

This log appears on Stackdriver (Fluentd output in GKE) as INFO log and like:

23:02:32.000 17-Oct-2017;04:36:29.744 : [main] [server:] [id:] [yt:] ERROR no.myproject.service.Server - call failed for some reason

So

23:02:32.000

is added to it (which is the normal behavior of Stackdriver). I will refer to this format as format 2.

As this log message actually is an ERROR log message (based on its payload content) I want it to appear as ERROR in Stackdriver (Fluentd).

I am trying:

<filter reform.**>
  type parser
  format /^(?<time>\d{2} [^\s]*) : (?<message2>[^ \]]*)\] (?<message3>[^ \]]*)\] (?<message4>[^ \]]*)\] (?<message5>[^ \]]*)\] (?<severity>\w)\s+(?<log2222>.*)/
  reserve_data true
  suppress_parse_error_log false
  key_name log
</filter>

hoping to get the severity of the message change to ERROR and also getting the content of the [..] fields in the log as some new key/values (in this case message2: main, etc).

But after adding this filter to my config file, the output logs are still as before and I don't see any change.

What am I missing? When I am writting my Regex pattern, I am not sure if I actually should consider the "log" field of the message in the log file in kubernetes or the one that I called format 2 (with the time added to it - on Stackdriver).

I would really appreciate any advice, it would be a great help.

-- samanta
fluentd
google-kubernetes-engine
kubernetes
logging
regex

Similar Questions

How to add external helm as part of my helm?
Turn on an API version for Google Container engine Cluster
Terraform GCE and kubernetes
gcloud kube-lego setup ssl issue

1 Answer

10/17/2017

I don't know the tool you're using, but it looks like first part of your regex does not match the exact format of the prefixed time in your log string.

Actually \d{2} will only match 2 digits.

To match the entire time prefix you may use (?:\d{2}:){2}\d{2}\.\d{3} instead.

One additional point regarding severity: you wrote (?<severity>\w) which captures only one word character. You may use (?<severity>\w+) to match several characters.

Your regex would then become:

^(?<time>(?:\d{2}:){2}\d{2}\.\d{3} [^\s]*) : (?<message2>[^ \]]*)\] (?<message3>[^ \]]*)\] (?<message4>[^ \]]*)\] (?<message5>[^ \]]*)\] (?<severity>\w+)\s+(?<log2222>.*)

That demo shoes a match.

-- PJProudhon
Source: StackOverflow