I have database migrations which I'd like to run before deploying a new version of my app into a Kubernetes cluster. I want these migrations to be run automatically as part of a Continuous Delivery pipeline. The migration will be encapsulated as a container image. What's the best mechanism to achieve this?
Requirements for a solution:
I had assumed that the Jobs functionality in Kubernetes would make this easy, but there appear to be a few challenges:
restartPolicy of never.Would using "bare pods" be a better approach? If so, how might that work?
You could try to make both the migration jobs and app independent of each other by doing the following:
Combining these two design approaches, you should be able to develop and execute the migration jobs and app independently of each other and not have to introduce any temporal coupling.
Whether this idea is actually reasonable to implement depends on more specific details of your case, such as the complexity of your database migration efforts. The alternative, as you mentioned, is to simply deploy unmanaged pods into the cluster that do the migration. This requires a bit more wiring as you will need to regularly check the result and distinguish between successful and failed outcomes.
blocking while waiting on the result of a queued-up job seems to require hand-rolled scripts
This isn't necessary anymore thanks to the kubectl wait command.
Here's how I'm running db migrations in CI:
kubectl apply -f migration-job.yml
kubectl wait --for=condition=complete --timeout=60s job/migration
kubectl delete job/migration
In case of failure or timeout, one of the two first CLI commands returns with an erroneous exit code which then forces the rest of the CI pipeline to terminate.
migration-job.yml describes a kubernetes Job resource configured with restartPolicy: Never and a reasonably low activeDeadlineSeconds.
You could also use the spec.ttlSecondsAfterFinished attribute instead of manually running kubectl delete but that's still in alpha at the time of writing and not supported by Google Kubernetes Engine at least.
Considering the age of this question I'm not sure if initContainers were available at the time but they are super helpful now.
https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
The way I recently set this up was to have a postgres pod and our django application running in the same namespace, however the django pod has 3 initContainers:
What this will do is run the django pod and the postgres pod in parallel but also continuously run the initContainers until the postgres pod comes up and then your migrations should run.
As for the pods perpetually restarting, maybe they've fixed the restartPolicy by now. I'm currently pretty new to kubernetes but this is what I've found works for me.