I have a fluentbit deployed as a sidecar. This fluentbit has an output of type Forward that is suppose to send the logs to a FluentD deployed as a DaemonSet.

The implementation works when using the PodIP of FluentD as host, but I get Connection refused when using the Service hostname from Kubernetes.

This is the error when using the Upstream approach:

[error] [net] TCP connection failed: fluentd.logging.svc.cluster.local:24224 (Connection refused)
[error] [net] socket #33 could not connect to fluentd.logging.svc.cluster.local:24224
[debug] [upstream] connection #-1 failed to fluentd.logging.svc.cluster.local:24224
[error] [output:forward:forward.0] no upstream connections available

This is the error when using the regular Host approach:

[error] [output:forward:forward.0] could not write forward header

I tried both using the Host parameter in Forward for Fluentbit, and also the Upstream functionality with the same outcome.

No network policies in place. This is the configuration with Upstream. With Host it will have Host and Port instead of Upstream in the OUTPUT section.

[SERVICE]
  Daemon Off
  Flush 5
  Log_Level debug
  Parsers_File parsers.conf
  Parsers_File custom_parsers.conf
  HTTP_Server Off
[INPUT]
  Name tail
  Path /var/app-logs/*
  Parser json
  Tag app-logs.*
  Mem_Buf_Limit 5MB
  Skip_Long_Lines On
[OUTPUT]
  Name forward
  Match app-logs.*
  Host fluentd.logging.svc.cluster.local
  Port 24244
[PARSER]
  Name json
  Format json
  Time_Key time
  Time_Format %Y-%m-%dT%H:%M:%S.%L

The FluentD deployment has a Service with the 24244 TCP port connected with the container TCP port 24244, where FluentD is listening.

A simple "nc" test also shows that I'm able to connect with the PodIP, but not to the Service hostname.

There's also an additional port in my FluenD daemonset which is for Prometheus metrics, and I can "nc" to that one using the host name.

This is the FluentD service

NAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)               AGE
fluentd   ClusterIP   10.102.255.48   <none>        24231/TCP,24244/TCP   4d6h

This is the FluentD deployment

Containers:
  fluentd:
    Container ID:   xxxx
    Image:          xxxx
    Image ID:       xxxx
    Ports:          24231/TCP, 24244/TCP
    Host Ports:     0/TCP, 0/TCP

This is the FluentD forward listener config

<source>
  @type forward
  port 24224
  bind 0.0.0.0
  @label @applogs
  tag applogs.*
</source>

Am I missing something obvious here?