Restart Kubernetes pod when change happens on another

3/30/2017

Two pods app and postgres are successfully created and are able to communicate through each other's services in a node. In the current process, the two pods get created at the same time, but that can be changed to have them be created/started in a sequence.

Initially, the database container in the postgres pod is empty and needs to be seeded. The seed process goes through the app pod and so, it needs to be up and running too. Once postgres is seeded, app still is not aware of this new data, and needs to be restarted. This is a flaw in app itself, that I have low control over.

Right now, the process is:

kubectl create -f pods.yaml       # creates `app` and `postgres` pods
kubectl exec app -- bash -c "<seed command>"
kubectl delete pod app
sleep 45                          # takes a while for `app` to terminate
kubectl create -f pods.yaml       # Ignore the "postgres pod and service already exist" error

Is there a better way of automatically co-ordinating a restart of app once postgres reaches a seeded state?

Perhaps there is some aspect/feature set of Kubernetes that I'm missing entirely which helps with such a circumstance....

-- changingrainbows
docker
kubernetes
minikube

2 Answers

4/2/2017

You can use a "readiness probe" on the postgresql pod that will not report the container as ready before the data is imported (e.g. query the DB or Table you import). You app container can query the readiness status of the db pod in order to restart automatically once it reports ready. The readiness probe can be a script performing the import. Here is an example (you need to replace the "SHOW DATABASES" command with whatever applies in your case):

spec:
  containers:
  - name: mysql
    image: mysql:latest
    ports:
    - containerPort: 3306
      name: mysql
    readinessProbe:
      exec:
        command:
        - /path-in-container/readiness-probe.sh
      initialDelaySeconds: 15
      timeoutSeconds: 5

readiness-probe.sh:

#!/bin/bash

MYSQL_USER="readinessProbe"
MYSQL_PASS="readinessProbe"
MYSQL_HOST="127.0.0.1"

mysql -u${MYSQL_USER} -p${MYSQL_PASS} -h${MYSQL_HOST} -e"SHOW DATABASES;"

if [ $? -ne 0 ]; then
  exit 1
else
  exit 0
fi

To read more on the topic refer to k8s docs:

readiness probes

health checking

-- Oswin Noetzelmann
Source: StackOverflow

3/30/2017

If the app doesn't need to run during the seeding process and you can make the seeding process idempotent then init containers can help you.

The is a good example available.

-- Janos Lenart
Source: StackOverflow