Writing a regex for a td agent

11/13/2019

Currently we have the following config for syslog tdagent(fluentd) config and would like to create another field for priority:Error for my log. How can I do this?

Log:

Nov 11 00:18:57 Build01v nagios: SERVICE ALERT: mmj21;Dropwizard MMJ Thread Pool;UNKNOWN;SOFT;1;**Error**: unable to access dropwizard metrics at localhost using port 8001

Current config:

<source>
  @type tail
    path           /var/log/messages
    pos_file       /var/log/td-agent/var_log_messages.pos
    read_from_head true
    tag            /var/log/messages
    <parse>
      @type regexp
        expression  ^(?<time>[^ ]* [^ ]* [^ ]*) (?<host>[^ ]*) (?<process>[^ ]*): (?<message>.*)$
        time_format %b %d %H:%M:%S
        time_key    time
    </parse>
</source>

Output:

https://fluentular.herokuapp.com/parse?regexp=%5E%28%3F%3Ctime%3E%5B%5E+%5D*+%5B%5E+%5D*+%5B%5E+%5D*%29+%28%3F%3Chost%3E%5B%5E+%5D*%29+%28%3F%3Cprocess%3E%5B%5E+%5D*%29%3A+%28%3F%3Cmessage%3E.*%29%24&input=Nov+11+00%3A18%3A57+Build01v+nagios%3A+SERVICE+ALERT%3A+mmj21%3BDropwizard+MMJ+Thread+Pool%3BUNKNOWN%3BSOFT%3B1%3B**Error%3A**+unable+to+access+dropwizard+metrics+at+localhost+using+port+8001&time_format=%25b+%25d+%25H%3A%25M%3A%25S

Records

Key     Value
host    Build01v
process nagios
message SERVICE ALERT: mmj21;Dropwizard MMJ Thread Pool;UNKNOWN;SOFT;1;**Error:** unable to access dropwizard metrics at localhost using port 8001
-- Kumar
fluentd
kubernetes
regex

1 Answer

11/13/2019

I'm guessing that maybe,

^(?<time>\S* \S* \S*) (?<host>\S*) (?<process>[^:]*): (?<priority>[^:]*):(?<message>.*)$

might be what you're trying to write.

RegEx Demo 1


If you wish to simplify/modify/explore the expression, it's been explained on the top right panel of regex101.com. If you'd like, you can also watch in this link, how it would match against some sample inputs.


RegEx Circuit

jex.im visualizes regular expressions:

enter image description here

-- Emma
Source: StackOverflow