Multiple regex matching in filebeat for message field

7/13/2021

I want to apply 2 regex expression with filebeat to drop events matching the content in message field.

I am able to make it work for single regex condition, but I am not sure how to configure multiple regex conditions.

regex list:

  • message: "(?i)cron"
  • message: "^now ([0-9]{4})-([0-1][0-9])-([0-3][0-9])\s([0-1][0-9]|[2][0-3]):([0-5][0-9]):([0-5][0-9])
    quot;

Following is the config I have done for single regex which will match "cron" case insensitive text anywhere in the message

- drop_event:
     when:
         regexp:
             message: "(?i)cron"

Refering to the Filebeat docs, I tried multiple configs but then filebeat won't startup:

  • Try 1:
- drop_event:
    or:
     - regexp:
        message: "(?i)cron"
     - regexp:
        message: "^now ([0-9]{4})-([0-1][0-9])-([0-3][0-9])\s([0-1][0-9]|[2][0-3]):([0-5][0-9]):([0-5][0-9])
quot;
  • Try 2:
- if:
    regexp:
      message: "(?i)cron"
  then:
    drop_event:
- if:
    regexp:
      message: "^now ([0-9]{4})-([0-1][0-9])-([0-3][0-9])\s([0-1][0-9]|[2][0-3]):([0-5][0-9]):([0-5][0-9])
quot;
then: drop_event:
-- Dishant Pandya
elasticsearch
filebeat
kibana
kubernetes
regex

1 Answer

7/15/2021

Figured Out How we can apply multiple filter using or operator in filebeat. I was close in the second attempt in the post. When is required, after that we can use whatever operator we like or and etc.

Here's example of how I am using it

processors:
  - drop_event.when:
     or:
     - contains:
         container.name: "nginx"
     - contains:
         container.name: "mongo"
     - contains:
         container.name: "mysql"
     - contains:
         container.name: "redis"
     - equals:
         container.name: "tecnativa/tcp-proxy"

  - drop_event.when:
     or:
     - regexp:
         message: "(?i)cron"
     - regexp:
         message: "In On Child added message"
     - regexp: 
         message: "In on Child removed message"
     - regexp:
         message: "then Moment"
     - regexp: 
         message: "call_duration"
     - regexp: 
         message: "now Moment"
     - regexp: 
         message: "CHAT NOTIFICATION CODE"
-- Dishant Pandya
Source: StackOverflow