Is there a way to limit the number of deployments a Kubernetes cluster will implement at once? With rolling deployments and 100% uptime, it's possible that updating all deployments at once could overload the nodes.
I know it is possible to limit the number of pods deployed per-namespace, but i was wondering if it is also possible to limit simultaneous deployments in a similar way. Say, for example, maximum of 10 deployments at once.
I could probably script out a limit to the number of deployments I send to the k8s API at once, but it would be nice if there was a setting I could use instead.
I have not tried this, but I assume that one can do this with an Admission controller. Set up an ExternalAdmissionWebhook:
https://kubernetes.io/docs/admin/extensible-admission-controllers/#external-admission-webhooks
When it receives an admissionReview request for a Deployment object, check the count and state of Deployments through the API, and reject the request if the criteria for concurrent Deployments are exceeded.
This solution can be helpful
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 50%
template:
---Setup RollingUpdate in your deployment.
rollingUpdate:
maxUnavailable: 50%
// The maximum number of pods that can be unavailable during the update.
// Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%).
// Absolute number is calculated from percentage by rounding down.
// This can not be 0 if MaxSurge is 0.
// Defaults to 25%.
// Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods
// immediately when the rolling update starts. Once new pods are ready, old RC
// can be scaled down further, followed by scaling up the new RC, ensuring
// that the total number of pods available at all times during the update is at
// least 70% of desired pods.In this way you can limit simultaneous deployments.
If you set resource requests on all your pods k8 will queue pods if resources are saturated
The first thing coming to my mind is to use resource limits and requests to make sure you're not overloading the cluster. This way, even if you update all the deployments, some pods will be in "pending" state until other deployments are successfully updated.