K
Q

Question

GKE: Sending File logs from docker container to Google Cloud Logging

6/27/2021

I'm trying to capture some logs that are file-based in an application pod on GKE and view them from Google Cloud Logging.

For various reasons, these application logs are not sent to STDOUT or STDERR (since those logs are automatically sent to Cloud Logging). I have been suggested to implement a scripting solution that tails the logs and sends them to STDOUT. However I was hoping in a side-car approach with Fluentd (or Fluentbit) logging agent that'll tail the logs and output them to Cloud Logging.

Using the sidecar image "k8s.gcr.io/fluentd-gcp:1.30", I've tried out the below YAML file (containing the fluentd configmap and deployment) below:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-log-config
data:
  fluentd.conf: |
    <source>
      type tail
      format none
      path /var/log/execution*.log
      pos_file /var/log/execution.pos
      tag app.*
    </source>

    <match **>
      type google_cloud
    </match>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app.kubernetes.io/name: app
    app.kubernetes.io/instance: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: app
      app.kubernetes.io/instance: app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app
        app.kubernetes.io/instance: app
    spec:
      serviceAccountName: app
      volumes:
        - name: executionlogs
          emptyDir: {}
        - name: fluentdconfig
          configMap:
            name: app-log-config
      containers:
        - name: app
          image: appimage:version
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: executionlogs
              mountPath: /tmp/executionLogs
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
        - name: log-agent
          image: "k8s.gcr.io/fluentd-gcp:1.30"
          imagePullPolicy: IfNotPresent
          env:
            - name: FLUENTD_ARGS
              value: "-c /etc/fluentd-config/fluentd.conf"
          volumeMounts:
            - name: executionlogs
              mountPath: /var/log
            - name: fluentdconfig
              mountPath: /etc/fluentd-config

Initially, the sidecar logs were throwing a 403 error, since I hadn't given the service account the requisite permissions (I was using GKE workload identity and the corresponding GCP IAM service account required logWriter permissions to be added). After fixing the error, I got the following logs:

2021-06-27 12:49:09 +0000 [info]: fluent/supervisor.rb:471:read_config: reading config file path="/etc/fluentd-config/fluentd.conf"
2021-06-27 12:49:09 +0000 [info]: fluent/supervisor.rb:337:supervise: starting fluentd-0.12.29
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-google-cloud' version '0.5.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-kafka' version '0.3.1'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-mongo' version '0.7.15'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-record-reformer' version '0.8.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-s3' version '0.7.1'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-scribe' version '0.10.14'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-systemd' version '0.0.5'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-td' version '0.10.29'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluentd' version '0.12.29'
2021-06-27 12:49:09 +0000 [info]: fluent/agent.rb:129:add_match: adding match pattern="**" type="google_cloud"
2021-06-27 12:49:10 +0000 [info]: plugin/out_google_cloud.rb:519:block in detect_platform: Detected GCE platform
2021-06-27 12:49:10 +0000 [info]: plugin/out_google_cloud.rb:290:configure: Logs viewer address: https://console.developers.google.com/
project/projectname/logs?service=compute.googleapis.com&key1=instance&key2=9071465168741286442
2021-06-27 12:49:10 +0000 [info]: fluent/root_agent.rb:147:add_source: adding source type="tail"
2021-06-27 12:49:10 +0000 [info]: fluent/engine.rb:133:configure: using configuration file: <ROOT>
  <source>
    type tail
    format none
    path /var/log/execution*.log
    pos_file /var/log/execution.pos
    tag app.*
  </source>
  <match **>
    type google_cloud
  </match>
</ROOT>
2021-06-27 12:52:10 +0000 [info]: plugin/in_tail.rb:557:initialize: following tail of /var/log/execution1.log
2021-06-27 12:53:10 +0000 [info]: plugin/out_google_cloud.rb:451:block in write: Successfully sent to Google Cloud Logging API.

Despite the successful message, I don't see anything on the Cloud Logging end.

So, here are my questions: 1. Is this there a better solution for my use-case? 2. Is the sidecar image I should be using? I wasn't able to find any other fluentd images and the one I'm using is 3 years old. I'd prefer to use something recommended by Google rather than create my own. 3. What do I additionally need to do to see the logs on Cloud Logging? How do I debug this further?

Thanks!

-- Sayon Roy Choudhury

fluentd

google-cloud-logging

google-kubernetes-engine

kubernetes

sidecar

2 Answers

6/27/2021

I do not see conf for filter, parser, input or output. There should be output conf with data like - OUTPUT Name stdout Match * Check more details here - https://docs.fluentd.org/output/stdout https://docs.fluentd.org/input/tail

-- subudear

Source: StackOverflow

7/2/2021

I’ve tried implementing the configuration you’ve implemented and faced the same issue. I then configured all sources to stream output to STDOUT and was able to view logs on the Cloud Logging dashboard.

Below are the sample configurations I’ve used.

Sample_map-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluentd.conf: |
    <source>
      type tail
      format none
      path /var/log/1.log
      pos_file /var/log/1.log.pos
      tag count.format1
    </source>

    <source>
      type tail
      format none
      path /var/log/2.log
      pos_file /var/log/2.log.pos
      tag count.format2
    </source>

    <match **>
      type stdout
    </match>

Sample-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        echo "$(date) INFO $i" >> /var/log/2.log;
        i=$((i+1));
        sleep 1;
      done      
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-agent
    image: k8s.gcr.io/fluentd-gcp:1.30
    env:
    - name: FLUENTD_ARGS
      value: -c /etc/fluentd-config/fluentd.conf
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: config-volume
      mountPath: /etc/fluentd-config
  volumes:
  - name: varlog
    emptyDir: {}
  - name: config-volume
    configMap:
      name: fluentd-config

-- Ashish

Source: StackOverflow

KQ

GKE: Sending File logs from docker container to Google Cloud Logging

Similar Questions

2 Answers

K
Q