Rancher 2.6.0: error="this plugin 'Fluent::Plugin::SplunkHecOutput' cannot handle arguments for <buffer ...> section"

9/29/2021

We are using the below tech stack rancher 2.6.0 K8s cluster created using rancher: v1.21.5 Separate Splunk container: Splunk Enterprise (Version: 8.2.2)

I installed a logging chart and configured the output/cluster output to Splunk using its token, index, and source. however, the fluentd-configcheck pods in rancher are failing because of below error.

fluentd -c /fluentd/etc/fluent.conf --dry-run
2021-09-29 08:10:12 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-datadog' version '0.13.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.4'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.16'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-kafka' version '0.16.3'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.8'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.2.5'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.7.2'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.7'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluentd' version '1.12.4'
2021-09-29 08:10:12 +0000 [info]: starting fluentd-1.12.4 as dry run mode ruby="2.7.3"
2021-09-29 08:10:12 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="this plugin 'Fluent::Plugin::SplunkHecOutput' cannot handle arguments for <buffer ...> section" 

Any help will be appreciated...

Below the generated config files

devnull.conf ---------
<label @ERROR>
<match **>
    @type null
    @id main-fluentd-error
</match>
</label>

<match **>
    @type null
    @id main-no-output
</match>

---------------------------
fluent.conf -------------------------

# include other config files
@include /fluentd/etc/input.conf
@include /fluentd/etc/generated.conf
@include /fluentd/etc/devnull.conf
@include /fluentd/etc/fluentlog.conf

----------------------------------------
generated.conf 
<source>
  @type forward
  @id main_forward
  bind 0.0.0.0
  port 24240
</source>
<match **>
  @type label_router
  @id main
  metrics false
  <route>
    @label @d1068204e7ff0cba866d5886a7a10f33
    metrics_labels {"id":"flow:cattle-logging-system:splunkflow"}
    <match>
      namespaces cattle-logging-system
      negate false
    </match>
  </route>
</match>
<label @d1068204e7ff0cba866d5886a7a10f33>
  <match **>
    @type splunk_hec
    @id flow:cattle-logging-system:splunkflow:output:cattle-logging-system:splunkoutput
    hec_host 172.27.1.119
    hec_port 8088
    hec_token 7376611a-e249-42a8-a57b-0a0bacfadc07
    index devopsci
    insecure_ssl true
    protocol http
    source index-devopsci
    <buffer tag,time>
      @type file
      chunk_limit_size 8MB
      path /buffers/flow:cattle-logging-system:splunkflow:output:cattle-logging-system:splunkoutput.*.buffer
      retry_forever true
      timekey 10m
      timekey_wait 10m
    </buffer>
  </match>
</label>

------------------------------------------
input.conf ------------------

# Enable RPC endpoint (this allows to trigger config reload without restart)
<system>
  rpc_endpoint 127.0.0.1:24444
  log_level info
  workers 1
</system>

# Prometheus monitoring
-----------------------------------------------
-- SHC
fluentd
kubernetes
rancher
splunk

0 Answers