Looks like journald pod logging ('kubectl logs' command) doesn't work with kubernetes v1.18.x. Is there a way to make this work with v1.18.x? I've set up a multinode cluster with docker logging driver as 'journald' (using /etc/docker/daemon.json) and using systemd-journal-gatewayd with pull approach to aggregate historical logs. However, I'm very much interested in being able to tail current logs using 'kubectl logs' or, 'kubectl logs -l 'app=label' --prefix -f' (for cluster-wide logs).
# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
# docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:20:16 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:23:58 2018
OS/Arch: linux/amd64
Experimental: false
# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
Running 'kubectl logs' gives the following error:
# kubectl logs <pod-name>
failed to try resolving symlinks in path "/var/log/pods/default_<pod-name>_xxxx/<container-name>/0.log": lstat /var/log/pods/default_<pod-name>_xxxx/<container-name>/0.log: no such file or directory
After you changed logging driver, the location of logs has changed to host's journal. So far the logs can be retrieved by docker but you have to let kubelet know about the changes.
You can do it by passing --log-dir=/var/log
to kubelet.
After adding the flag you have run systemctl daemon-reload
and restart kubelet. It has to be done on all of the nodes.