We're moving a legacy app to Kubernetes. We will have many instances of it running (a Kubernetes namespace per customer), so we want to automate our application upgrade process.
Kubernetes has well established patterns for rolling upgrades, but I can't use them (yet). My application requires the following process: 1) all existing pods are deleted 2) a database upgrade job (Kubernetes Job) runs and completes successfully 3) new pods are created
We define the pods through a Deployment. The database upgrade job is idempotent as long as we never run more than one at a time.
I assume my workflow is not an uncommon one for legacy applications, and yet I can't find any established patterns or tools preconfigured. Is there something out there? If I do have to write my own operator (or use something like Kudo), what is the best set of steps for it to perform?
Yes, there is an existing process for that:
kubectl scale command to scale down the existing Deployment to zero replicas: kubectl scale --replicas=0 deploy/my-legacy-deploymentWait for that to stabilize (there's your requested downtime ;-)
Using kubectl wait will be helpful, although I personally don't have experience introducing downtime in order to know the remaining arguments to wait to suggest here
You can also use something like while true; do [[ 0 -eq $(kubectl get pods -o name -l legacy-deployment-selector | wc -l) ]] && break; done to stall until there are no pods remaining
Run your database job, or migrations of your choosing
Deploy the new version as you normally would; depending on the tool you use, this may or may not actually influence the currently zero-scaled Deployment
For example, kubectl set image deploy/my-legacy-deployment "*=example.com/my/new/image" will leave the Deployment at zero replicas
But a helm install --upgrade legacy whatever-else may very well set the Deployment's replicas to the value found in the chart
If your tool has not yet scaled up the new Deployment, you can now set it back to the desired value with the opposite command: kubectl scale --replicas=3 deploy/my-legacy-deployment