How to roll back a deployment to a previous revision automatically when deployment has crossed 'progressDeadlineSeconds'?

5/24/2019

I'm working on deploying a docker image in kubernetes. The first time I deployed the container, I used:

kubectl apply -f <deployment_file>.yaml

and the container was successfully deployed in a pod.

Also, the deployment_file looks something like this:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: publisher
spec:
  replicas: 2
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 300
  progressDeadlineSeconds: 900
  template:
    metadata:
      labels:
        app: publisher
    spec:
      containers:
      - name: publisher
        image: 123dev.azurecr.io/publisher:{{ci-build-number}}
        env:
          - name: ENVIRONMENT
            value: "dev"
        ports:
        - containerPort: 8080

I have defined the 'progressDeadlineSeconds' attribute in the yaml file above.

To keep a track of the deployment, I used:

kubectl rollout status deployment.v1beta1.apps/publisher

Now if I want to update the container with a new image, I can again reuse the command

kubectl apply -f <deployment_file>.yaml

to apply the update.

But what if applying the update fails due to some reason (let's say the docker image is corrupt), is there a way to automatically trigger a rollback to the previous revision when - pods status is not set to 'running' OR the execution time crosses 'pregressDeadlineSeconds'?

Till now I haven't found a way to execute a rollback automatically. Thoughts would be appreciated.

-- Swastik Gupta
azure-aks
docker
kubernetes

2 Answers

5/24/2019

follow the below steps.

1. append the --record param to deployment, as shown below
kubectl apply -f <deployment_file>.yaml --record

2. kubectl rollout history deploy <deployment-name> to check deployment history. for example,
kubectl rollout history deploy mynginx
deployments "mynginx"
REVISION  CHANGE-CAUSE
3         kubectl set image deploy mynginx mynginx=nginx:1.12.2
4         kubectl set image deploy mynginx mynginx=nginx:1.13.8


3. you can rollback to previous version using revision, say to revision 3
kubectl rollout undo deploy mynginx --to-revision=3
-- P Ekambaram
Source: StackOverflow

1/2/2020

After the kubectl apply command you can check if the deployment rolled out successfully or not and then, if necessary, the kubectl rollout undo command can rollback to the previous revision. Also, you can use the sleep Linux command to wait some time before that.

On Linux:

sleep 900 && \
if [[ $(timeout 5 kubectl rollout status -f <deployment_file>.yaml) != *"successfully rolled out"* ]]; then     
    kubectl rollout undo -f <deployment_file>.yaml
fi > /dev/null 2>&1 &

timeout 5 stops the kubectl rollout status if its stuck after 5 seconds.

> /dev/null 2>&1 & runs the command in the background and without terminal output.

-- Ben Even Tsur
Source: StackOverflow