Change kubernetes (kubeadm) logging away from /var/log/messages

8/8/2018

we have a kubernetes cluster running on Centos 7. However all logging is going to /var/log/messages which is making centos system logs hard to read. Is there a way I can tell kubeadm/kubernetes to log to /var/log/kubernetes rather?

We are already sending our application (pod) logs to a mountpoint. We need to move the stderr logs of kubernetes.

-- warhansen
kubernetes
logging

1 Answer

8/8/2018

However all logging is going to /var/log/messages which is making centos system logs hard to read. Is there a way I can tell kubeadm/kubernetes to log to /var/log/kubernetes rather?

No, not exactly, but you can reconfigure Docker to log in a different way.

This might depend on the Docker version you're running but in my CentOS 7 VM (a couple of weeks old) i'm running Docker version 1.13.1, installed via yum.

When looking through the docs for version 1.13 and the latest stable version of Docker they say more or less the same thing:

If you do not specify a logging driver, the default is json-file.

The version of Docker i installed via yum had the following line in an environment file (/etc/sysconfig/docker) that is loaded when starting Docker:

OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'

As you can see the logging driver are configured as journald, that should be the reason you're seeing logs from your containers in /var/log/messages. You can check which logging drive are configured with:

docker info | grep 'Logging Driver'

The logging driver decides where all of the logs, in Docker meaning stderr and stdout from containers are sent. Docker supports a couple of different logging drivers, if you choose to configure e.g. json-file which might be the best choice if you want to relocate the logging from an OS perspective ("changing" the log path). Every Docker container will have it's own log written to /var/log/pods/<ID>/<NAME>/<LOGFILE>, actually the log files are symlinks back to /var/lib/docker/containers/<ID>/<ID>-json.log.

If you do configure json-file then remove the --log-driver=journald flag and instead configure this in the /etc/docker/daemon.json file, mentioned in the docs. With json-file you can configure things like log rotation and log file sizes, please consult the docs for more options.

When configuring via the daemon.json file this becomes a global setting, you can always override the logging driver used for a specific container with docker run ... --log-driver.

These logging changes applies for everything running within Docker, to move logging for e.g. kubelet which runs alongside Docker on your host you can look at the configurable options. Default the kubelet stderr logs are logged via journald and ends up in /var/log/messages, to change this behavior can add the --log-dir options and point to another location.

In the end of the day i think it's good to give log shipping a thought and investigate the other logging drivers if they might fit into your environment.

-- mikejoh
Source: StackOverflow