Kubernetes Automated Undo Rollbacks in Azure DevOps rolls back to too old version

12/12/2019

I am using Azure DevOps Release Pipelines for my AKS cluster. My process is, in short, building my Docker Image, pushing it to ACR (Azure Container Registry) in my CI Pipeline. In my CD Pipeline I proceed to Apply Deployment with this Container.

If any task within the Release Pipeline, I rollback to the previous deployment as the assumption is that something went wrong.

My issue I found today is the following scenario:

CI Pipeline builds and pushed Image ID: 333 to ACR
CD Pipelines Apply Deployment Image ID: 333 to K8s

I am happy and see that the Rollhout History has added another Revision.

I trigger another CD Release of the same artifact and it is applied again.

I check the Rollout History and see the Revision is not changed.

I do another deploy of the CD Pipeline with the same Artifact the next day. Another task fails in my pipeline due some external dependency (In my case someone changed a Postman test which causes them to fail) My pipeline Rollbacks to an old Image ID: 222

I am now left with an old image running in my Cluster, even though the Image ID: 333 was fine all along.

I could only rollback in case Deployment Verification fails, i.e. my Deployment fails to roll out properly onto my cluster.

But if I were to push a new image which would roll out fine but something inside it is bad, which would be detected by my Postman calls, then I would like to rollback to previous working image.

In other words, I have two scenarios where I want it to rollback due to Postman (external task) failing, and sometimes I do not want it to.

How can I circumvent this? Since apply Deployment does not update my Revision History when the deployment.yaml file has not changed it causes it to rollback to old versions.

-- Lud1212
azure
azure-devops
docker
kubernetes

0 Answers