Fluentd DaemonSet is forwarding my container logs using in_tail method to an elasticsearch cluster in AWS.

When I list my pods:

-bash-4.2$ kubectl get pods |grep fluentd
fluentd-d8vbp                       1/1       Running   0          2h
fluentd-h6mk6                       1/1       Running   0          2h
fluentd-nrp8l                       1/1       Running   0          2h

The pos file looks like this, in each of these pods:

-bash-4.2$ kubectl exec -it fluentd-d8vbp -- ls -al /var/log/fluentd-containers.log.pos
-rw-r--r-- 1 root root **2516** Jun 28 05:27 /var/log/fluentd-containers.log.pos
-bash-4.2$ kubectl exec -it fluentd-h6mk6 -- ls -al /var/log/fluentd-containers.log.pos
-rw-r--r-- 1 root root **0** Jun 28 02:44 /var/log/fluentd-containers.log.pos
-bash-4.2$ kubectl exec -it fluentd-nrp8l -- ls -al /var/log/fluentd-containers.log.pos
-rw-r--r-- 1 root root **0** Jun 28 02:44 /var/log/fluentd-containers.log.pos

You can see that only one file has data in it (2516), and other two are empty(0). Is this an expected behavior?

Another issue is that whenever we add a node, say at 10:00 AM, the fluentd will stop forwarding the logs until we re-create the pods. The serious issue is that we will miss the logs till then. Say we recreate the pods at 10:15 AM, we will miss the logs from 10:00 AM - 10:15 AM.

My yaml file:

<source>
      @type tail
      @id xxx_logs
      path /var/log/containers/xxx*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format regexp
          expression /myregexp1/
        </pattern>
        <pattern>
          format regexp
          expression /myregexp2/
        </pattern>
      </parse>
    </source>

Anyone facing similar issue?