We have a k8 cluster. I am trying to access logs from inside and kubectl won't work inside. Where would be the logs stored in k8? We do not have systemd and found in the docs that:
If systemd is not present, they write to .log files in the /var/log directory. System components inside containers always write to the /var/log directory, bypassing the default logging mechanism.But I could not find any logs in here. So how can I get access to these logs which I would get by kubectl logs from inside the pod? How does default logging work in k8 without any logging mechanism setup?
PS: I did go through other similar posts and had no luck with those.
I understood that the application just logs in a file/directory inside the container.
Could you use
kubectl exec -it <podname> bash or
kubectl exec -it <podname> sh
to enter the container/pod and check the logs inside the container.
If you have the logs you want to access available via kubectl logs then it means they are ultimately output to stdout or stderr. Docker and kubelet work on top of standard outputs, processing these logs in their own fashion (ie. by logging plugins). When your process throws something to sdtout, it is obviously not stored anywhere on the local filesystem of the container.
That said, you can configure your app to log to files, but mind that you need to handle logrotation, cleanup etc. or your container will perpetualy grow. If you can't have both in parallel, you do loose them from docker/kubernetes logs though, which is not so nice. If that is the case, you can have a process in a sidecar that reads logfiles from mounted volume and sends them to stdout/stderr.
The real question is why do you need to access logs inside POD. Knowing that maybe there is a better way to achieve what you need (ie. pipe them first via some parser process).
If the application does not log to a file, it may log to stdout sometimes (which kubectl logs <pod name> should also show).
docker logs <name or ID of the container/var/log directory does not persist in a volume mounted in the container, it will be lost when the pod restarts or moves in the cluster as the /var/log directory will be ephemeral. Check if the pod has restarted or moved in the cluster.Find if the pod uses any volumes for persistent storage of /var/log by doing:
kubectl get <pod name> -o yaml | grep -i volume
kubectl get persistentvolumes --all-namespaces
kubectl get persistentvolumeclaims --all-namespaces