As part of our CI pipeline, we have a deployment script for a number of web services that looks something like this:
kubectl apply -f deployment1.yml
kubectl apply -f deployment2.ymlThe problem is that the next stage of the pipeline sometimes fails because the services are not ready by the time it starts.
I would like to add a line in the script that says something like:
Wait until all deployments are in the Ready state, or fail if more than 30 seconds has elapsed.I thought that the following would work but unfortunately it seems that the timeout flag is not available:
kubectl rollout status deployment deployment1 --timeout=30s
kubectl rollout status deployment deployment2 --timeout=30sI don't want to run "kubectl rollout status" without a timeout as that will cause our build to hang if there is a failure in one of the deployments.
I found a solution that works well. Set the property .spec.progressDeadlineSeconds to a value such as 30 (default is 600 or ten minutes) and kubectl rollout status deployment will wait for this amount of time before displaying an error message and exiting with a non zero exit code:
$ kubectl rollout status deploy/nginx
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
error: deployment "nginx" exceeded its progress deadlineDocumentation is here: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#failed-deployment
You could possibly handle this with pure bash i.e.:
ATTEMPTS=0
ROLLOUT_STATUS_CMD="kubectl rollout status deployment/myapp -n namespace"
until $ROLLOUT_STATUS_CMD || [ $ATTEMPTS -eq 60 ]; do
$ROLLOUT_STATUS_CMD
ATTEMPTS=$((attempts + 1))
sleep 10
done
This is specified in this blog
However, I do not believe there is a Kubernetes Native way to wait for a deployments rollout, you could possibly accheive this with hooks in Helm or Webhooks if you want to get really fancy