K
Q

Docker cleaning up Kubernetes Pod logs too quickly

1/27/2020

I am running a number of CronJobs to automate tasks on my home server. When a Job runs and its Pods complete, I can view logs using kubectl logs <podName>, however after a certain amount of time (less than 5mins but more than 30s) the Pod still exists in Kubernetes but I get:

unable to retrieve container logs for docker://<containerID>

I have not enabled any log rotation myself so I assume Docker is cleaning up the logs once the container finishes. How can I configure Docker to retain these container logs so that I can see the Jobs logs after a longer period of time (say 1 day) after the Job completes.

  • Cluster provisioner: kubeadm 1.17.0
  • Kubernetes version: 1.17.0
  • Docker version: 18.06.3-ce
  • uname -r: 4.19.86-coreos
-- dippynark
docker
kubernetes

Similar Questions

kubernetes role to hide the master
How to execute the master go scritpt in kubernetes
How run a kubernetes job to configure a statefulset (e.g., a DB)
How to handle K8S CRD CRUD requests by only one instance of an operator

2 Answers

1/27/2020

Went for a simple CronJob using docker prune

docker system prune -f -a --filter "until=168h"
docker volume prune -f
-- dippynark
Source: StackOverflow
1/27/2020

You can try setting up minimum-container-ttl-duration flag in Kubelet. This is the minimum age for a finished container before it is garbage collected.

Also in the job spec you can specify a TTL

apiVersion: batch/v1
kind: Job
metadata:
  name: pi-with-ttl
spec:
  ttlSecondsAfterFinished: 100

The Job pi-with-ttl will be eligible to be automatically deleted, 100 seconds after it finishes.

-- Arghya Sadhu
Source: StackOverflow