K
Q

Why does my deployment terminate pods on deployment?

2/14/2022

I have a rolling update deployment strategy with a 0 max unavailable value. Unfortunately when I deploy many pods get terminated and I am not sure why.

Example:

ScalingReplicaSet Scaled down replica set deployment-xyz to 27
ScalingReplicaSet Scaled down replica set deployment-xyz to 10

It went from 27 to 10 upon deployment

Strategy:

StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 25% max surge

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: app
  name: app
spec:
  progressDeadlineSeconds: 600
  replicas: 25
  selector:
    matchLabels:
      component: app
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        component: app
    spec:
      containers:
      - args:
        - command
        image: image:tag
        imagePullPolicy: IfNotPresent
        name: app

The HPA:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: app
spec:
  maxReplicas: 200
  minReplicas: 25
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app

Where could I look to understand why this is happening?

-- Adnan Mujkanovic
kubernetes

Similar Questions

How to inject AWS account number into k8s ServiceAccount?
Kubernetes doesn't recognise wildcard*
When creating a AzureFile persistent volume via a dynamic persistent volume claim the AzureFile created has no metadata
Load balance Kafka consumer multiple instances
How to wait for a rollout to complete in Kubernetes

1 Answer

2/14/2022

Okay, I have replicated your scenario by deploying 25 replicas of nginx and then triggering a rolling restart using:

kubectl patch deployment test-deployment \
  -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

This is the output:

Events:
  Type    Reason             Age                 From                   Message
  ----    ------             ----                ----                   -------
  Normal  ScalingReplicaSet  4m18s               deployment-controller  Scaled up replica set test-deployment-5c876b9587 to 25
  Normal  ScalingReplicaSet  82s                 deployment-controller  Scaled up replica set test-deployment-6765d87dcf to 7
  Normal  ScalingReplicaSet  80s                 deployment-controller  Scaled down replica set test-deployment-5c876b9587 to 24
  Normal  ScalingReplicaSet  80s                 deployment-controller  Scaled up replica set test-deployment-6765d87dcf to 8
  Normal  ScalingReplicaSet  80s                 deployment-controller  Scaled down replica set test-deployment-5c876b9587 to 23
  Normal  ScalingReplicaSet  80s                 deployment-controller  Scaled up replica set test-deployment-6765d87dcf to 9
  Normal  ScalingReplicaSet  80s                 deployment-controller  Scaled down replica set test-deployment-5c876b9587 to 22
  Normal  ScalingReplicaSet  80s                 deployment-controller  Scaled up replica set test-deployment-6765d87dcf to 10
  Normal  ScalingReplicaSet  80s                 deployment-controller  Scaled down replica set test-deployment-5c876b9587 to 21
  Normal  ScalingReplicaSet  77s (x16 over 80s)  deployment-controller  (combined from similar events): Scaled down replica set test-deployment-5c876b9587 to 11

It replicates your scenario almost exactly. Notice how the first two lines are:

Scaled up replica set test-deployment-5c876b9587 to 25
Scaled up replica set test-deployment-6765d87dcf to 7

What it's doing here is scaling up the initial set of 25 (after I deployed the deployment). The second line is when I triggered the update. It scales up 7 because 25% of 25 (your maxSurge value) is 6.25, so it rounds up to 7.

You can then see the rollout happening as it updates one pod at a time (terminating one pod from the old set, and bringing it up on the new set), since you specified 0 maxUnavailable, it won't terminate more than it's bringing up in the new replica set.

Looking at your original output:

ScalingReplicaSet Scaled down replica set deployment-xyz to 27
ScalingReplicaSet Scaled down replica set deployment-xyz to 10

I am not seeing any massive jumps in "scaled down" messages myself. Other than the last one where it jumps from 21 to 11, but it also says "Combined from similar events", meaning it's hidden messages from the event log because they are similar.

Are you able to share the exact event message log? (You can redact the deployment name if you wish, but keep the hashes visible as it's then possible to distinguish between the old and new replicasets)

UPDATE

As mentioned in the comments, OP was using Helm to deploy the changes, and that was setting replicas to 10, which caused the pods to drop from 25 to 10, then the HPA kicks in and scales it back up to 25, and then terminates those as the rollout progresses.

My recommendation is to make the HPA minReplicas AND the deployment replicas equal (use the same variable for both). Then you won't get caught out by this in future.

-- Blender Fox
Source: StackOverflow