I was playing around with Kubernetes on AWS with t2.medium EC2 instances having 20GB of disk space and one of the nodes ran out of disk space after a few days. It seems to be caused by a combination of Docker images and logs.
From what I've read, Kubernetes has its own Docker GC to manage Docker's disk usage, and log rotation. I'm guessing 20GB is not enough for Kubernetes to self-manage disk usage. What's a safe disk size for a production environment?
When following the standard installation with the GKE as described in the quickstart guide you'll end up with 3 x n1-standard-1 nodes (see machine types) with 100 GB of storage per node.
Looking at the nodes right after the cluster creation then gives you these numbers for the diskspace:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 1.2G 455M 767M 38% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmp 1.9G 24K 1.9G 1% /tmp
run 1.9G 684K 1.9G 1% /run
shmfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 95G 2.4G 92G 3% /var
/dev/sda8 12M 28K 12M 1% /usr/share/oem
media 1.9G 0 1.9G 0% /media
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 256K 0 256K 0% /mnt/disks
tmpfs 1.0M 120K 904K 12% /var/lib/cloud
overlayfs 1.0M 124K 900K 13% /etcThese numbers might give you a starting point, but as others have pointed out already, the rest depends on your specific requirements.
I think there is no general answer. You have already answered yourself- you need enough space for docker images, containers and logs. How much- it depends on number of images used, containers running, amount of logs generated and amount of logs you want to keep.
Kubernetes-based applications should not create log files on pod volume. Logs should go either to shared volume (external to pod) or to log management database like elasticSearch, graylog. Pods will be created and deleted many times during cluster lifetime. Log files mounted on pod volume will disappear when pod is gone. Most likely, you want your log to live longer than that.
From experiment you already did you should know those numbers, and be able to estimate usage for your needs.
See https://kubernetes.io/docs/admin/garbage-collection/ for more details.