I have a go based application deployed on a kubernetes cluster.
I am using golang:1.13
as my base image
The /var/log
folders on those containers keep filling up causing the pods to evict.
Is there a way where I could specify something in deployment.yaml or something like that where I disable system logs from being written to file system?
You should probably have some kind of log rotation system set up, most installers do it. But failing that, if you want to disable collection of container output to log files you can do that in your Dockerd configuration, usually it's set up in the systemd unit file for dockerd but the specifics depend on your deployment.
There are a few ways to sort this out and solving this issue really depends on your application, environment and your creativity.
The Kubernetes documentation describes a few ways to achieve what you need and on all of them the use of logrotate is in place.
Suggestions are:
As an example on how you can use your creativity to sort this out, a simple solution would be to mount a volume on /var/log and save all your logs externally, this way you can have more space to store these logs and you can deal with them externally, cleaning or maintaining as you wish. In this example I'm using a CronJob to "delete" the files.
Example:
apiVersion: v1
kind: PersistentVolume
metadata:
name: logsvol
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: logsvol
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
name: counter
spec:
containers:
- name: count
image: busybox
args:
- /bin/sh
- -c
- >
i=0;
while true;
do
echo "$i: $(date)" >> /var/log/1.log;
i=$((i+1));
sleep 1;
done
volumeMounts:
- name: varlog
mountPath: /var/log
volumes:
- name: varlog
persistentVolumeClaim:
claimName: logsvol
You can create a CronJob to clean (in this example I'm just listing it) these logs for you:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: logcleaner
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox
args:
- /bin/sh
- -c
- ls -l /var/log
volumeMounts:
- name: varlog
mountPath: /var/log
volumes:
- name: varlog
persistentVolumeClaim:
claimName: logsvol
restartPolicy: OnFailure