I am trying to push the logs to splunk from Kubernetes using fluentd as a daemonset.

When I set format none and push to splunk it works. But I want to add kubernetes_metadata so I updated to format json and added the kubernetes_metadata filter.

Then, I get a 400 Bad Request

{"text": "Invalid data format", "code": 6, "invalid-event-number": 0 }

Here is my fluent.conf file

# Ignore fluentd log messages
<match fluent.**>
  @type null
</match>

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /fluentd/log/docker-containers.log.pos
  tag kubernetes.*
  format json
  read_from_head true
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

<match kubernetes.**>
  @type splunk-http-eventcollector
  server <host>:8088
  protocol https
  verify false
  token ***
  source fluentd-kube-containers
  sourcetype _json
  host "#{ENV['HOSTNAME']}"
  buffer_chunk_limit 700k
  batch_size_limit 1000000
  buffer_type file
  buffer_path /fluentd/log/fluentd-buffer
  flush_interval 10s
</match>