How to do controlled rollout using Kubernetes deployment

4/14/2021

We have 1000 store nodes and need to deploy an application image on every kubernetes node by rolling out in the below order and would like to specify the deployment node details during the deployment. Is there a way to specify node details in the command line when we execute kubectl create or apply deployment commands?

This application image would be configured to store/node specific details during container/POD creation.

1 node on day 1, 10 node on day 2, 100 node on day 3 etc.

-- kalaiventhan
kubernetes
kubernetes-pod

1 Answer

4/26/2021

Answering on the question from the title:

How to do controlled rollout using Kubernetes deployment

You can create a Deployment that will have specific fields in its manifest that will configure the way Kubernetes handles it.

With the fields like: podAntiAffinity, requiredDuringSchedulingIgnoredDuringExecution you can ensure that Kubernetes will distribute the Pods equally across the cluster Nodes. You can read more about it by following below documentation:

Having in mind following rollout schedule:

DAYREPLICAS_COUNT
11
210
3100
41000

You could use CI/CD tools (like for example Jenkins) to rollout (change) the amount of replicas of your Deployment across a specific schedule.

You could create a Jenkins pipeline with a deploy stage where you could put your own command with it's scheduler (or delay).

The example of such Deployment that could be used with Jenkins is following:

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: ${REPLICAS_COUNT}
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - nginx
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: nginx
        image: nginx
EOF

This Deployment will assign Pods to the Nodes that aren't already having an already running replica of this Deployment (i.e. 1 Pod = 1 Node). If the amount of Pods exceeds the amount of Nodes they will remain in Pending state.


Additional resources:

-- Dawid Kruk
Source: StackOverflow