I'm using the following config to have fluentd read the auth.logs and send it to elastic search but i'm faced with an error saying pattern doesn't match and the logs are not pushed to ES.
I'm using the pattern defined in fluentd syslog parser plugin rfc3164-pattern
<source>
@type tail
path /var/log/auth.log
pos_file /var/log/auth.pos
format /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\]) *(?<message>.*)$/
tag authlog
</source>
<match authlog.**>
@type elasticsearch
hosts "ESHOST:PORT"
logstash_format true
logstash_prefix "server-authlogs"
include_tag_key true
flush_interval 5s
logstash_dateformat %Y.%m.%d
time_precision 3
</match>
Output Error:
2019-04-16 08:00:50 +0000 [warn]: #0 pattern not match: "Apr 16 08:00:50 hostname-1415 sshd[15134]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)" 2019-04-16 08:00:50 +0000 [warn]: #0 pattern not match: "Apr 16 08:00:50 hostname-1415 systemd-logind[1138]: New session 10 of user ubuntu."
For those who are looking for something similar, here is the config that works well.
<source>
type tail
path /var/log/foo/auth.log
pos_file /var/log/auth.pos
tag authlog
format /^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
</source>
<match authlog.**>
@type elasticsearch
hosts "ESHOST:PORT"
logstash_format true
logstash_prefix "server-authlogs"
include_tag_key true
flush_interval 5s
logstash_dateformat %Y.%m.%d
time_precision 3
</match>
For a auth.log pattern of:
Apr 16 18:02:02 host-1415 sshd[11111]: Accepted password for ubuntu from 111.11.111.11 port 11111 ssh2How about using parser_syslog? If /var/log/auth.log has syslog format(RFC3164).
<source>
@type tail
path /var/log/auth.log
pos_file /var/log/auth.pos
tag authlog
<parse>
@type syslog
message_format rfc3164
with_priority false
</parse>
</source>