How can one define log retention for kubernetes pods? For now it seems like the log file size is not limited, and it is uses the host machine complete resources.
According to Logging Architecture from kubernetes.io there are some options
First option
Kubernetes currently is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that. For example, in Kubernetes clusters, deployed by the kube-up.sh script, there is a logrotate tool configured to run each hour. You can also set up a container runtime to rotate application’s logs automatically, e.g. by using Docker’s log-opt. In the kube-up.sh script, the latter approach is used for COS image on GCP, and the former approach is used in any other environment. In both cases, by default rotation is configured to take place when log file exceeds 10MB.
Also
Second option
Sidecar containers can also be used to rotate log files that cannot be rotated by the application itself. An example of this approach is a small container running logrotate periodically. However, it’s recommended to use stdout and stderr directly and leave rotation and retention policies to the kubelet.
You can always set the logging retention policy on your docker nodes
See: https://docs.docker.com/config/containers/logging/json-file/#examples
I've just got this working by changing the ExecStart line in /etc/default/docker and adding the line --log-opt max-size=10m
Please note, that this will affect all containers running on a node, which makes it ideal for a Kubernetes setup (because my real-time logs are uploaded to an external ELK stack)