Kubernetes Automated Rollbacks

12/4/2018

I was trying to find some alternative for docker-swarm rollback command, which allows you to specify rollback strategy in the deployment file. In k8s ideally it should use readinessProbe, and if didn't pass failureThreshold it should rollback, before starting deployment of the next pod (to avoid downtime).

Currently, in my deployment script, I'm using hook kubectl rollout status deployment $DEPLOYMENT_NAME || kubectl rollout undo deployment $DEPLOYMENT_NAME, which works, but it's not ideal because first rollout command will trigger error way after unhealthy pod will be deployed and a healthy one will be destroyed which will cause downtime.

Ideally, it shouldn't even kill current pod before a new one will pass readinessProbe

-- Sarkis Arutiunian
kubernetes

1 Answer

12/4/2018

There is no specific rollback strategy in a Kubernetes deployment. You could try a combination of RollingUpdate with max unavailable (aka Proportional Scaling), then at some point pause your deployment resume if everything looks good and then rollback if something went wrong.

The recommended way is really to use another deployment as canary split the traffic through a load balancer between canary and non-canary, then if everything goes well upgrade the non-canary and shut down the canary. If something goes wrong shutdown the canary and keep the non-canary until the issue is fixed.

Another strategy is to use something like Istio that facilitates canary deployments.

-- Rico
Source: StackOverflow