I was so frustrated from finding no solution to this seemingly common problem that I built a docker image that tails log files and sends them to stdout, to be used as a sidecar container.

Here's what I did:

1. I added a volume with emptyDir{} to the pod
2. I mounted that volume to my main container, with the mountPath being the directory to which it writes the logs to
3. I added another container to the pod, called "logger", with the image being the log tracker I wrote (lutraman/logger-sidecar:v2), and mounted the same volume to /logs (I programmed the script to read the logs from this directory)

then, all the logs written to that directory, can be accessed by kubectl logs <pod-name> -c logger

Here is an example yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy
  labels:
    app: dummy
spec:
  selector:
    matchLabels:
      app: dummy
  template:
    metadata:
      labels:
        app: dummy
    spec:
      volumes:
        - name: logs
          emptyDir: {}
      containers:
        - name: dummy-app # the app that writes logs to files
          image: lutraman/dummy:v2
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          env:
            - name: MESSAGE
              value: 'hello-test'
            - name: LOG_FILE
              value: '/var/log/app.log'
          volumeMounts:
            - name: logs
              mountPath: /var/log
        - name: logger # the sidecar container tracking logs and sending them to stdout
          image: lutraman/logger-sidecar:v2
          volumeMounts:
            - name: logs
              mountPath: /logs

For anyone who is interested, here is how I made the sidecar container:

Dockerfile:

FROM alpine:3.9

RUN apk add bash --no-cache

COPY addTail /addTail
COPY logtrack.sh /logtrack.sh

CMD ["./logtrack.sh"]

addTail:

#!/bin/sh

(exec tail -F logs/$3 | sed "s/^/$3: /" ) &
echo $! >> /tmp/pids

logtrack.sh:

#!/bin/bash

trap cleanup INT

function cleanup() {
  while read pid; do kill $pid; echo killed $pid; done < /tmp/pids
}

: > /tmp/pids

for log in $(ls logs); do
  ./addTail n logs $log
done

inotifyd ./addTail `pwd`/logs:n