I have recently set up redis via https://github.com/tarosky/k8s-redis-ha, this repo includes an init container, and I have included an extra init container in order to get passwords etc set up.

I am seeing some strange (and it seems undocumented) behavior, whereby the init containers run as expected before the redis container starts, however then they run subsequently every hour, close to an hour. I have tested this behavior using a busybox init container (which does nothing) on deployments & statefulset and experience the same behavior, so it is not specific to this redis pod.

I have tested this on bare metal with k8s 1.6 and 1.8 with the same results, however when applying init containers to GKE (k8s 1.7) this behavior does not happen. I can't see any flags for GKE's kubelet to dictate this behavior.

See below for kubectl describe pod showing that the init containers are run when the main pod has not exited/crashed.

Name:           redis-sentinel-1
Namespace:      (redacted)
Node:           (redacted)/(redacted)
Start Time:     Mon, 12 Mar 2018 06:20:55 +0000
Labels:         app=redis-sentinel
                controller-revision-hash=redis-sentinel-7cc557cf7c
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"(redacted)","name":"redis-sentinel","uid":"759a3a3b-25bd-11e8-a8ce-0242ac110...
                security.alpha.kubernetes.io/unsafe-sysctls=net.core.somaxconn=1024
Status:         Running
IP:             (redacted)
Controllers:    StatefulSet/redis-sentinel
Init Containers:
  redis-ha-server:
    Container ID:       docker://557d777a7c660b062662426ebe9bbf6f9725fb9d88f89615a8881346587c1835
    Image:              tarosky/k8s-redis-ha:sentinel-3.0.1
    Image ID:           docker-pullable://tarosky/k8s-redis-ha@sha256:98e09ef5fbea5bfd2eb1858775c967fa86a92df48e2ec5d0b405f7ca3f5ada1c
    Port:
    State:              Terminated
      Reason:           Completed
      Exit Code:        0
      Started:          Tue, 13 Mar 2018 03:01:12 +0000
      Finished:         Tue, 13 Mar 2018 03:01:12 +0000
    Ready:              True
    Restart Count:      0
    Environment:        <none>
    Mounts:
      /opt from opt (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hkj6d (ro)
  -redis-init:
    Container ID:       docker://18c4e353233a6827999ae4a16adf1f408754a21d80a8e3374750fdf9b54f9b1a
    Image:              gcr.io/(redacted)/redis-init
    Image ID:           docker-pullable://gcr.io/(redacted)/redis-init@sha256:42042093d58aa597cce4397148a2f1c7967db689256ed4cc8d9f42b34d53aca2
    Port:
    State:              Terminated
      Reason:           Completed
      Exit Code:        0
      Started:          Tue, 13 Mar 2018 03:01:25 +0000
      Finished:         Tue, 13 Mar 2018 03:01:25 +0000
    Ready:              True
    Restart Count:      0
    Environment:        <none>
    Mounts:
      /opt from opt (rw)
      /secrets/redis-password from redis-password (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hkj6d (ro)
Containers:
  redis-sentinel:
    Container ID:       docker://a54048cbb7ec535c841022c543a0d566c9327f37ede3a6232516721f0e37404d
    Image:              redis:3.2
    Image ID:           docker-pullable://redis@sha256:474fb41b08bcebc933c6337a7db1dc7131380ee29b7a1b64a7ab71dad03ad718
    Port:               26379/TCP
    Command:
      /opt/bin/k8s-redis-ha-sentinel
    Args:
      /opt/sentinel.conf
    State:              Running
      Started:          Mon, 12 Mar 2018 06:21:02 +0000
    Ready:              True
    Restart Count:      0
    Readiness:          exec [redis-cli -p 26379 info server] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      SERVICE:          redis-server
      SERVICE_PORT:     redis-server
    Mounts:
      /opt from opt (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hkj6d (ro)
  redis-sword:
    Container ID:       docker://50279448bbbf175b6f56f96dab59061c4652c2117452ed15b3a5380681c7176f
    Image:              tarosky/k8s-redis-ha:sword-3.0.1
    Image ID:           docker-pullable://tarosky/k8s-redis-ha@sha256:2315c7a47d9e47043d030da270c9a1252c2cfe29c6e381c8f50ca41d3065db6d
    Port:
    State:              Running
      Started:          Mon, 12 Mar 2018 06:21:03 +0000
    Ready:              True
    Restart Count:      0
    Environment:
      SERVICE:          redis-server
      SERVICE_PORT:     redis-server
      SENTINEL:         redis-sentinel
      SENTINEL_PORT:    redis-sentinel
    Mounts:
      /opt from opt (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hkj6d (ro)
Conditions:
  Type          Status
  Initialized   True
  Ready         True
  PodScheduled  True
Volumes:
  opt:
    Type:       HostPath (bare host directory volume)
    Path:       /store/redis-sentinel/opt
  redis-password:
    Type:       Secret (a volume populated by a Secret)
    SecretName: redis-password
    Optional:   false
  default-token-hkj6d:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-hkj6d
    Optional:   false
QoS Class:      BestEffort
Node-Selectors: <none>
Tolerations:    <none>
Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath                           Type            Reason  Message
  ---------     --------        -----   ----                    -------------                           --------        ------  -------
  20h           30m             21      kubelet, 10.1.3.102     spec.initContainers{redis-ha-server}    Normal          Pulling pulling image "tarosky/k8s-redis-ha:sentinel-3.0.1"
  21h           30m             22      kubelet, 10.1.3.102     spec.initContainers{redis-ha-server}    Normal          Started Started container
  21h           30m             22      kubelet, 10.1.3.102     spec.initContainers{redis-ha-server}    Normal          Created Created container
  20h           30m             21      kubelet, 10.1.3.102     spec.initContainers{redis-ha-server}    Normal          Pulled  Successfully pulled image "tarosky/k8s-redis-ha:sentinel-3.0.1"
  21h           30m             22      kubelet, 10.1.3.102     spec.initContainers{redis-init}         Normal          Pulling pulling image "gcr.io/(redacted)/redis-init"
  21h           30m             22      kubelet, 10.1.3.102     spec.initContainers{redis-init}         Normal          Pulled  Successfully pulled image "gcr.io/(redacted)/redis-init"
  21h           30m             22      kubelet, 10.1.3.102     spec.initContainers{redis-init}         Normal          Created Created container
  21h           30m             22      kubelet, 10.1.3.102     spec.initContainers{redis-init}         Normal          Started Started container

Note the Containers in the pod which started at Mon, 12 Mar 2018 06:21:02 +0000 (with 0 restarts) and the Init Containers which started from Tue, 13 Mar 2018 03:01:12 +0000. These seem to re-run every hour pretty much in an interval of hour.

Our bare metal must be misconfigured for init containers somewhere? Can anyone shed any light on this strange behavior?