K
Q

Cron in the same container works locally but fails in cluster

5/2/2017

I have a super simple container with cron scheduled:

* * * * * root /bin/bash /alive.sh

alive.sh:

#!/bin/bash

/bin/echo "I'm alive"
/bin/echo $(/bin/date) >> /tmp/alive.log

I build the docker image locally and run it:

docker build -t orian/crondemo:v0 .
docker run --rm -it --name crondemo orian/crondemo:v0

And after a minute or so I can check that a new file is being created:

docker exec crondemo ls /tmp

The I tag and push the image to Google Container Registry:

TAG=eu.gcr.io/<PROJECT_ID>/crondemo:v0
docker tag orian/crondemo:v0 $TAG
kubectl docker -- push $TAG

Starting a pod manually:

kubectl run crondemo --image=$TAG --replicas=1 --restart=Never

And verifying that it works:

kubectl exec crondemo ls /tmp

And here is a problem, a /tmp/alive.log file is not being written to. Where is the problem?

I've prepared a repo with sample: https://github.com/orian/k8s-cron-demo

Notice

  • I've also tested overwriting /var/spool/cron/crontabs/root but it didn't solve the problem.
  • I'm using docker image: openjdk:8-jre. Before switching I used alpine and crond. It's seemed to work then.

Edit2 - found (this is crazy):

-- Paweł Szczur
cron
docker
google-kubernetes-engine
kubernetes

Similar Questions

Why is kubernetes reporting scheduling error because of Memory resource limits?
Kubernetes: Cinder volume not working
Kubernetes internal load balancer on gcloud
Exclude path from Google Cloud Endpoints

1 Answer

5/4/2017

I've followed a https://stackoverflow.com/a/21928878/436754 on enabling logs.

Running: /var/log/syslog

May  4 12:33:05 crondemo rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="14" x-info="http://www.rsyslog.com"] start
May  4 12:33:05 crondemo rsyslogd: imklog: cannot open kernel log(/proc/kmsg): Operation not permitted.
May  4 12:33:05 crondemo rsyslogd-2145: activation of module imklog failed [try http://www.rsyslog.com/e/2145 ]
May  4 12:33:08 crondemo cron[38]: (CRON) INFO (pidfile fd = 3)
May  4 12:33:08 crondemo cron[39]: (CRON) STARTUP (fork ok)
May  4 12:33:08 crondemo cron[39]: (*system*) NUMBER OF HARD LINKS > 1 (/etc/crontab)
May  4 12:33:08 crondemo cron[39]: (*system*crondemo) NUMBER OF HARD LINKS > 1 (/etc/cron.d/crondemo)
May  4 12:33:08 crondemo cron[39]: (CRON) INFO (Running @reboot jobs)
May  4 12:34:01 crondemo cron[39]: (*system*) NUMBER OF HARD LINKS > 1 (/etc/crontab)
May  4 12:34:01 crondemo cron[39]: (*system*crondemo) NUMBER OF HARD LINKS > 1 (/etc/cron.d/crondemo)

This made me Google for cron "NUMBER OF HARD LINKS > 1" and I've found: https://github.com/phusion/baseimage-docker/issues/198

The workaround is to modify a Dockerfile to overwrite the cron file on start instead being mounted by Docker.

  • Dockerfile COPY cronfile /cronfile
  • docker-entrypoint.sh: cp /cronfile /etc/cron.d/crondemo

A branch with workaround: https://github.com/orian/k8s-cron-demo/tree/with-rsyslog

-- Paweł Szczur
Source: StackOverflow