K
Q

Question

kubernetes fluentd how to exclude liveness, readiness /health check and fluent.trace?

7/14/2019

I am using the lastest Fluentd /td-agent. How to filter out liveness, readiness health check from Fluentd streaming to Elasticsearch? I would also like to exclude fluent.trace. Sample output seen from Kibana:

{
  "_index": "aspnetapistarter.logs-2019.07.14",
  "_type": "_doc",
  "_id": "zFSL72sBodUyUY6Nychc",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-07-14T08:13:59.1083593+00:00",
    "level": "Information",
    "messageTemplate": "{HostingRequestStartingLog:l}",
    "message": "Request starting HTTP/1.1 GET http://10.1.1.55:5000/health/live  ",
    "fields": {
      "Protocol": "HTTP/1.1",
      "Method": "GET",
      "ContentType": null,
      "ContentLength": null,
      "Scheme": "http",
      "Host": "10.1.1.55:5000",
      "PathBase": "",
      "Path": "/health/live",
      "QueryString": "",
      "HostingRequestStartingLog": "Request starting HTTP/1.1 GET http://10.1.1.55:5000/health/live  ",
      "EventId": {
        "Id": 1
      },
      "SourceContext": "Microsoft.AspNetCore.Hosting.Internal.WebHost",
      "RequestId": "0HLO86B34CTB4:00000001",
      "RequestPath": "/health/live",
      "CorrelationId": null,
      "ConnectionId": "0HLO86B34CTB4"
    },
    "renderings": {
      "HostingRequestStartingLog": [
        {
          "Format": "l",
          "Rendering": "Request starting HTTP/1.1 GET http://10.1.1.55:5000/health/live  "
        }
      ]
    },
    "tag": "aspnetcore"
  },
  "fields": {
    "@timestamp": [
      "2019-07-14T08:13:59.108Z"
    ]
  },
  "sort": [
    1563092039108
  ]
}

I have tried this but doesn't work:

    <filter fluent.*>
      @type grep
      <exclude>
        key tag
        pattern fluent.trace
     </exclude>
    </filter>
    <filter aspnetcore-access>
      @type grep
      <exclude>
        key fields.Path
        pattern health
     </exclude>
    </filter>
    <filter aspnetcore-access>
      @type grep
      <exclude>
        key fields.RequestPath
        pattern health
     </exclude>
    </filter>

Thanks. WTF is this: "It looks like your post is mostly code; please add some more details."

-- Kok How Teh

elasticsearch

filter

fluentd

kubernetes

logging

1 Answer

7/14/2019

The exact syntax is here:

<filter foo.bar>
  @type grep
  <regexp>
    key message
    pattern /cool/
  </regexp>
  <regexp>
    key hostname
    pattern /^web\d+\.example\.com$/
  </regexp>
  <exclude>
    key message
    pattern /uncool/
  </exclude>
</filter>

https://docs.fluentd.org/filter/grep

So for example you can try:

<filter aspnetcore-access>
      @type grep
      <exclude>
        key message
        pattern /health/
     </exclude>
    </filter>

-- Ijaz Ahmad Khan

Source: StackOverflow

KQ

kubernetes fluentd how to exclude liveness, readiness /health check and fluent.trace?

Similar Questions

1 Answer

K
Q