As a part of debug i need to track down events like pod creation and removal. in my kubernetes set up I am using logging level 5.
Kube api server, scheduler, controller, etcd are running on master node and the minion nodes is running kubelet and docker.
I am using journalctl to get K8s logs on master node as well on worker node. On worker node i can see logs from Docker and Kubelet. These logs contain events as i would expect as i create and destroy pods.
However on Master node i dont see any relevant logs which may indicate a pod creation or removal request handling.
what other logs or method i can use to get such logs from Kubernetes master components (API server, controller, scheduler, etcd)?
i have checked the logs from API server, controller, scheduler, etcd pods; they dont seem to have such information.
thanks
System component logs:
There are two types of system components:
those that run in a container
and those that do not run in a container.
For example:
The Kubernetes scheduler and kube-proxy run in a container
The kubelet and container runtime, for example Docker, do not run in containers.
On machines with systemd, the kubelet and container runtime write to journald. If systemd is not present, they write to .log files in the /var/log directory. System components inside containers always write to the /var/log directory, bypassing the default logging mechanism. They use the klog logging library.
Master components logs:
Get them from those containers running on master nodes.
$
$ docker ps | grep apiserver
d6af65a248f1 af20925d51a3 "kube-apiserver --ad…" 2 weeks ago Up 2 weeks k8s_kube-apiserver_kube-apiserver-minikube_kube-system_177a3eb80503eddadcdf8ec0423d04b9_0
5f0e6b33a29f k8s.gcr.io/pause-amd64:3.1 "/pause" 2 weeks ago Up 2 weeks k8s_POD_kube-apiserver-minikube_kube-system_177a3eb80503eddadcdf8ec0423d04b9_0
$
$
$ docker logs -f d6a But all of this approach to logging is just for testing , you should stream all the logs , ( app logs , container logs , cluster level logs , everything) to a centeral logging system such as ELK or EFK.