Restarting completed k8 job pods after the aws cluster restart

8/4/2020

I have a k8s job pod which sends rest call to the micro service to initialize the redis when we deploy the service with kubectl apply. But when the aws cluster restarts, then the k8s job pod is not getting restated as it's in completed status and only micro service pods are getting restarted.

How can I restart completed job pod with aws cluster restart?

-- sushma
kubernetes
kubernetes-pod

1 Answer

8/5/2020

Kubernetes was designed for high availibility, to work without failures and downtime so "restarting kubernetes cluster" is not a common usecase. Although I uderstand that in some cases you may want to shut down a development cluster for weekends to save money and hope that when you start it, it will come back to its previous state.

Jobs cannot be restarted once completed, so this is not a good option.

You could use a deployment and run your initializing script (the script that sends the rest calls) as initcontainer and after it finishes run pause container (so it doesn't use resources).

So now when you restart the cluser, initialization pod will be scheduled and the init container will run. Just be aware that in this case when the node on which the pod is running goes down, the pod will get rescheduled and your initializing pod will be run again. This looks like a reasonable option but you need to make sure that the rest calls are idempotent.

Other thing you could do is to use redis oparator to handle the configuration for you. Find redis operators on this list of operators.

-- Matt
Source: StackOverflow