Fluentd is unable to write the logs to /fluentd/log Directory

10/16/2020

I have deployed a fluentd sidecar container with my application in a pod to collect logs from my app.

Here's my sidecar manifest sidecar.yaml:

spec:
  template:
    spec:
      containers:
        - name: fluentd
          image: fluent/fluentd
          ports:
          - containerPort: 24224
            protocol: TCP
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          terminationMessagePath: /dev/termination-log
          volumeMounts:
          - mountPath: /etc/td-agent/config.d
            name: configmap-sidecar-volume
          securityContext:
            runAsUser: 101
            runAsGroup: 101

I used this manifest and patched it to my deployment using the following command:

kubectl patch deployment my-deployment --patch “$(cat sidecar.yaml)

The deployment was successfully updated, however the my fluentd container can't seem to start and is throwing the following error:

2020-10-16 09:07:07 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-10-16 09:07:08 +0000 [info]: gem 'fluentd' version '1.11.2'
2020-10-16 09:07:08 +0000 [warn]: [output_docker1] 'time_format' specified without 'time_key', will be ignored
2020-10-16 09:07:08 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="out_file: `/fluentd/log/docker.20201016.log` is not writable"

This is my fluent.conf file:

<source>
  @type forward
  bind 127.0.0.1
  port 24224
  <parse>
    @type json
  </parse>
</source>

What is causing this issue?

-- bakadevops
fluentd
kubernetes

1 Answer

10/16/2020

fluentd's UID default to 1000 unless it changed via env FLUENT_UID.

/fluentd/log/docker.20201016.log is not writable - error says that your user 101 doesn't have write permission to the log file. Change the security context to 1000 or set env FLUENT_UID=101 to fix the issue.

spec:
  template:
    spec:
      containers:
        - name: fluentd
          image: fluent/fluentd
          ports:
          - containerPort: 24224
            protocol: TCP
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          terminationMessagePath: /dev/termination-log
          volumeMounts:
          - mountPath: /etc/td-agent/config.d
            name: configmap-sidecar-volume
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000

Related resources:

-- Kamol Hasan
Source: StackOverflow