Kubernetes with logrotate sidecar mount point issue

10/15/2018

I am trying to deploy a test pod with nginx and logrotate sidecar. Logrotate sidecar taken from: logrotate

My Pod yaml configuration:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-apache-log
  labels:
    app: nginx-apache-log

spec:
  containers:

  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
    volumeMounts:
    - name: logs
      mountPath: /var/log

  - name: logrotate
    image: path/to/logrtr:sidecar
    volumeMounts:
    - name: logs
      mountPath: /var/log

  volumes:
  - name: logs
    emptyDir: {}

What I'd like to achieve is Logrotate container watching /var/log//.log, however with the configuration above, nginx container is failing because there is no /var/log/nginx:

nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
2018/10/15 10:22:12 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (2: No such file or directory)

However if I change mountPath for nginx from

mountPath: /var/log 

to:

mountPath: /var/log/nginx

then it is starting, logging to /var/log/nginx/access.log and error.log, but logrotate sidecar sees all logs in /var/log not /var/log/nginx/. It is not a problem with just one nginx container, but I am planning to have more container apps logging to their own /var/log/appname folders.

Is there any way to fix/workaround that? I don't want to run sidecar for each app.

If I change my pod configuration to:

  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
    volumeMounts:
    - name: logs
      mountPath: /var/log
  initContainers:
    - name: install
      image: busybox
      command:
      - mkdir -p /var/log/nginx
      volumeMounts:
      - name: logs
        mountPath: "/var/log"

then it is failing with:

Warning  Failed     52s (x4 over 105s)  kubelet, k8s-slave1  Error: failed to start container "install": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"mkdir -p /var/log/nginx\": stat mkdir -p /var/log/nginx: no such file or directory": unknown
-- Syn Romana
kubernetes
logrotate

2 Answers

10/15/2018

If you are running nginx in kubernetes, it is probably logging to stdout. When you run kubectl logs <nginx pod> nginx it will show you access and error logs. These logs are automatically logrotated by kubernetes, so you will not need a logrotate sidecar in this case.

If you are ever running pods that are not logging to stdout, this is a bit of an antipattern in kubernetes. It is more to your advantage to always log to stdout: kubernetes can take care of log rotation for you, and it is also easier to see logs with kubectl logs than by running kubectl exec and rummaging around in a running container

-- Lindsay Landry
Source: StackOverflow

10/15/2018

Leave the mount path as /var/log. In your nginx container, execute mkdir /var/log/nginx in a startup script. You might have to tweak directory permissions a bit to make this work.

-- Roland Weber
Source: StackOverflow