Is there an easy way to disable linux logs on kubernetes pod containers?

5/29/2020

I have a go based application deployed on a kubernetes cluster.

I am using golang:1.13 as my base image

The /var/log folders on those containers keep filling up causing the pods to evict.

Is there a way where I could specify something in deployment.yaml or something like that where I disable system logs from being written to file system?

-- developer747
dockerfile
kubernetes

2 Answers

5/29/2020

You should probably have some kind of log rotation system set up, most installers do it. But failing that, if you want to disable collection of container output to log files you can do that in your Dockerd configuration, usually it's set up in the systemd unit file for dockerd but the specifics depend on your deployment.

-- coderanger
Source: StackOverflow

5/29/2020

There are a few ways to sort this out and solving this issue really depends on your application, environment and your creativity.

The Kubernetes documentation describes a few ways to achieve what you need and on all of them the use of logrotate is in place.

Suggestions are:

As an example on how you can use your creativity to sort this out, a simple solution would be to mount a volume on /var/log and save all your logs externally, this way you can have more space to store these logs and you can deal with them externally, cleaning or maintaining as you wish. In this example I'm using a CronJob to "delete" the files.

Example:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: logsvol
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logsvol
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi    
---
apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        i=$((i+1));
        sleep 1;
      done            
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    persistentVolumeClaim:
      claimName: logsvol

You can create a CronJob to clean (in this example I'm just listing it) these logs for you:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: logcleaner
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - ls -l /var/log
            volumeMounts:
            - name: varlog
              mountPath: /var/log
          volumes:
          - name: varlog
            persistentVolumeClaim:
              claimName: logsvol
          restartPolicy: OnFailure
-- mWatney
Source: StackOverflow