We are using the below tech stack rancher 2.6.0 K8s cluster created using rancher: v1.21.5 Separate Splunk container: Splunk Enterprise (Version: 8.2.2)
I installed a logging chart and configured the output/cluster output to Splunk using its token, index, and source. however, the fluentd-configcheck pods in rancher are failing because of below error.
fluentd -c /fluentd/etc/fluent.conf --dry-run
2021-09-29 08:10:12 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-datadog' version '0.13.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.4'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.16'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-kafka' version '0.16.3'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.8'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.2.5'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.7.2'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.7'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.1'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2021-09-29 08:10:12 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-09-29 08:10:12 +0000 [info]: gem 'fluentd' version '1.12.4'
2021-09-29 08:10:12 +0000 [info]: starting fluentd-1.12.4 as dry run mode ruby="2.7.3"
2021-09-29 08:10:12 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="this plugin 'Fluent::Plugin::SplunkHecOutput' cannot handle arguments for <buffer ...> section"
Any help will be appreciated...
Below the generated config files
devnull.conf ---------
<label @ERROR>
<match **>
@type null
@id main-fluentd-error
</match>
</label>
<match **>
@type null
@id main-no-output
</match>
---------------------------
fluent.conf -------------------------
# include other config files
@include /fluentd/etc/input.conf
@include /fluentd/etc/generated.conf
@include /fluentd/etc/devnull.conf
@include /fluentd/etc/fluentlog.conf
----------------------------------------
generated.conf
<source>
@type forward
@id main_forward
bind 0.0.0.0
port 24240
</source>
<match **>
@type label_router
@id main
metrics false
<route>
@label @d1068204e7ff0cba866d5886a7a10f33
metrics_labels {"id":"flow:cattle-logging-system:splunkflow"}
<match>
namespaces cattle-logging-system
negate false
</match>
</route>
</match>
<label @d1068204e7ff0cba866d5886a7a10f33>
<match **>
@type splunk_hec
@id flow:cattle-logging-system:splunkflow:output:cattle-logging-system:splunkoutput
hec_host 172.27.1.119
hec_port 8088
hec_token 7376611a-e249-42a8-a57b-0a0bacfadc07
index devopsci
insecure_ssl true
protocol http
source index-devopsci
<buffer tag,time>
@type file
chunk_limit_size 8MB
path /buffers/flow:cattle-logging-system:splunkflow:output:cattle-logging-system:splunkoutput.*.buffer
retry_forever true
timekey 10m
timekey_wait 10m
</buffer>
</match>
</label>
------------------------------------------
input.conf ------------------
# Enable RPC endpoint (this allows to trigger config reload without restart)
<system>
rpc_endpoint 127.0.0.1:24444
log_level info
workers 1
</system>
# Prometheus monitoring
-----------------------------------------------