I have a use-case for concurrent restart of all pods in a statefulset.
Does kubernetes statefulset support concurrent restart of all pods?
According to the statefulset documentation, this can be accomplished by setting the pod update policy to parallel as in this example:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql-db
spec:
podManagementPolicy: Parallel
replicas: 3
However this does not seem to work in practice, as demonstrated on this statefulset running on EKS:
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: producer
namespace: ragnarok
spec:
selector:
matchLabels:
app: producer
replicas: 10
podManagementPolicy: "Parallel"
serviceName: producer-service
template:
metadata:
labels:
app: producer
spec:
containers:
- name: producer
image: archbungle/load-tester:pulsar-0.0.49
imagePullPolicy: IfNotPresent
Rollout restart happens in sequence as if disregarding the rollout policy setting:
(base) welcome@Traianos-MacBook-Pro eks-deploy % kubectl get pods -n ragnarok | egrep producer
producer-0 1/1 Running 0 3m58s
producer-1 1/1 Running 0 3m56s
producer-2 1/1 Running 0 3m53s
producer-3 1/1 Running 0 3m47s
producer-4 1/1 Running 0 3m45s
producer-5 1/1 Running 0 3m43s
producer-6 1/1 Running 1 3m34s
producer-7 0/1 ContainerCreating 0 1s
producer-8 1/1 Running 0 16s
producer-9 1/1 Running 0 19s
(base) welcome@Traianos-MacBook-Pro eks-deploy % kubectl get pods -n ragnarok | egrep producer
producer-0 1/1 Running 0 4m2s
producer-1 1/1 Running 0 4m
producer-2 1/1 Running 0 3m57s
producer-3 1/1 Running 0 3m51s
producer-4 1/1 Running 0 3m49s
producer-5 1/1 Running 0 3m47s
producer-6 0/1 Terminating 1 3m38s
producer-7 1/1 Running 0 5s
producer-8 1/1 Running 0 20s
producer-9 1/1 Running 0 23s
(base) welcome@Traianos-MacBook-Pro eks-deploy % kubectl get pods -n ragnarok | egrep producer
producer-0 1/1 Running 0 4m8s
producer-1 1/1 Running 0 4m6s
producer-2 1/1 Running 0 4m3s
producer-3 1/1 Running 0 3m57s
producer-4 1/1 Running 0 3m55s
producer-5 0/1 Terminating 0 3m53s
producer-6 1/1 Running 0 4s
producer-7 1/1 Running 0 11s
producer-8 1/1 Running 0 26s
producer-9 1/1 Running 0 29s
As the document pointed, Parallel pod management will effective only in the scaling operations. This option only affects the behavior for scaling operations. Updates are not affected.
Maybe you can try something like
kubectl scale statefulset producer --replicas=0 -n ragnarok
and
kubectl scale statefulset producer --replicas=10 -n ragnarok
According to documentation, all pods should be deleted and created together by scaling them with the Parallel policy.
Reference : https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#parallel-pod-management