I'm setting up a Graylog service in my kubernetes cluster, and have installed Graylog Sidecars on my worker nodes, with NXLog backends, but I am unable to get the sidecars to send logs to the input I've configured (GELF HTTP). On the Sidecar status page, all the files and directories located under the /var/log directory I configured the sidecar to include in the sidecar.yaml conf file are listed, and they get updated regularly. This should mean the sidecar is connected in at least some way, but the logs still don't appear in my Input.

I've been able to send logs to my GELF HTTP input by routing it via my ingress and the curl command below:

curl -X POST -H 'Content-Type: application/json' -d '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' 'http://<host>/gelf'

#graylog ingress
...
  spec:
    rules:
    - host: <host>
      http:
        paths:
        - backend:
            serviceName: graylog-tcp
            servicePort: gelf1
          path: /gelf
        - backend:
            serviceName: graylog-web
            servicePort: graylog

Graylog configurations:

#NXLog Collector Configuration
define ROOT /usr/bin

<Extension gelfExt>
  Module xm_gelf
  # Avoid truncation of the short_message field to 64 characters.
  ShortMessageLength 65536
</Extension>

<Extension syslogExt>
  Module xm_syslog
</Extension>

User nxlog
Group nxlog

Moduledir /usr/libexec/nxlog/modules
CacheDir /var/spool/nxlog/data
PidFile /var/run/nxlog/nxlog.pid
LogFile /var/log/nxlog/nxlog.log
LogLevel INFO


<Input sidecar_files>
    Module im_file
    File '/var/log/graylog-sidecar/*.log'
    PollInterval 1
    SavePos True
    ReadFromLast True
    Recursive True
    RenameCheck False
    Exec $FileName = file_name(); # Send file name with each message
</Input>

<Output gelf_http>
    Module om_http
    URL https://<host>/gelf
    ContentType application/json
    <Exec>
      # These fields are needed for Graylog
      $gl2_source_collector = '${sidecar.nodeId}';
      $collector_node_id = '${sidecar.nodeName}';
    </Exec>
</Output>

<Route route-1>
  sidecar_files => gelf_http
</Route>

#GELF HTTP input
bind_address: 0.0.0.0
decompress_size_limit: 8388608
enable_cors: true
idle_writer_timeout: 60
max_chunk_size: 65536
number_worker_threads: 1
override_source: <empty>
port: 12201
recv_buffer_size: 425984
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password: ********

I would expect there to be either logs in my Input message stream, or error messages in my pod logs, but I am finding neither.