kubernetes: kubectl only streams a certain amount of logs

5/21/2019

I have created a pod that produces several millions of log lines.

However kubectl logs -f <pod_name> at some point stops and the output freezes.

Here is my experiment:

kubectl logs --follow=true test-logs-job-13263-t5dbx | tee file1
kubectl logs --follow=true test-logs-job-13263-t5dbx | tee file2
kubectl logs --follow=true test-logs-job-13263-t5dbx | tee file3

Each of the above processes was interrupted (Ctrl+C) as soon as I stopped receiving any input on my screen.

So:

$ wc -l file*
  106701 file1
  106698 file2
  106698 file3
  320097 total

It is worth mentioning also that each of these files contains different logs, i.e. the actual logs that were streamed at the time the kubectl logs -f command was actually executed.

Is there a setting that limits the max amount of logs to be streamed to the above number? (~10700)

My cluster is on GKE for what that matters.

edit: For some reason this seem to happen only on GKE; when I run the same experiment on another k8s cluster (katakoda) the logs were streamed without any issue whatsoever.

-- pkaramol
kubectl
kubernetes
logging

1 Answer

5/21/2019

Kubernetes does not provide logging itself but uses what the container engine provides. If you were running without kubernetes on a plain docker installation the default configuration of docker is to use the json-file driver and keep 10 mb of logs per container. See https://docs.docker.com/config/containers/logging/json-file/ If you need a huge amount of logs consider using central logging infrastructure and configure log forwarding. (This is supported by docker as well for several logging backends, see https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers )

There is a bug reported that log rotation can interrupt the streaming of kubectl logs --follow: https://github.com/kubernetes/kubernetes/issues/28369

-- Thomas
Source: StackOverflow