K
Q

Running a pod/container in Kubernetes that applies maintenance to a DB

7/16/2021

I have found several people asking about how to start a container running a DB, then run a different container that runs maintenance/migration on the DB which then exits. Here are all of the solutions I've examined and what I think are the problems with each:

  1. Init Containers - This wont work because these run before the main container is up and they block the starting of the main container until they successfully complete.
  2. Post Start Hook - If the postStart hook could start containers rather than simply exec a command inside the container then this would work. Unfortunately, the container with the DB does not (and should not) contain the rather large maintenance application required to run it this way. This would be a violation of the principle that each component should do one thing and do it well.
  3. Sidecar Pattern - This WOULD work if the restartPolicy were assignable or overridable at the container level rather than the pod level. In my case the maintenance container should terminate successfully before the pod is considered Running (just like would be the case if the postStart hook could run a container) while the DB container should Always restart.
  4. Separate Pod - Running the maintenance as a separate pod can work, but the DB shouldn't be considered up until the maintenance runs. That means managing the Running state has to be done completely independently of Kubernetes. Every other container/pod in the system will have to do a custom check that the maintenance has run rather than a simple check that the DB is up.
  5. Using a Job - Unless I misunderstand how these work, this would be equivalent to the above ("Separate Pod").
  6. OnFailure restart policy with a Sidecar - This means using a restartPolicy of OnFailure for the POD but then hacking the DB container so that it always exits with an error. This is doable but obviously just a hacked workaround. EDIT: This also causes problems with the state of the POD. When the maintenance runs and stays up and both containers are running, the state of the POD is Ready, but once the maintenance container exits, even with a SUCCESS (0 exit code), the state of the POD goes to NotReady 1/2.

Is there an option I've overlooked or something I'm missing about the above solutions? Thanks.

-- Jim
kubernetes
kubernetes-pod

Similar Questions

Istio Ingress does not work - only port forwarding works
Can we read environment variable on Azure Kubernetes Pod?
Kubernetes - run job after pod status is ready
Running scripts using lifecycle hook does not work properly in kubernetes
Time zone for moment in a pod

1 Answer

7/17/2021

One option would be to use the Sidecar pattern with 2 slight changes to the approach you described:

  1. after the maintenance command is executed, you keep the container running with a while : ; do sleep 86400; done command or something similar.
  2. You set an appropriate startupProbe in place that resolves successfully only when your maintenance command is executed successfully. You could for example create a file /maintenance-done and use a startupProbe like this:
startupProbe:
  exec:
    command:
    - cat
    - /maintenance-done
  initialDelaySeconds: 5
  periodSeconds: 5

With this approach you have the following outcome:

  1. Having the same restartPolicy for both your database and sidecar containers works fine thanks to the sleep hack.
  2. You Pod only becomes ready when both containers are ready. In the sidecar container case this happens when the startupProbe succeedes.

Furthermore, there will be no noticeable overhead in your pod: even if the sidecar container keeps running, it will consume close to zero resources since it is only running the sleep command.

-- whites11
Source: StackOverflow