We have a statefulset that we want to have minimum downtime (like any other statefulset out there I suppose), but the pod gets stuck at "terminating" state since the readiness probe failure threshold is 5 (adds a 5s downtime during "Terminating" state for no reason). So, for it to terminate faster, I reduced the failure threshold to 1, it now terminates faster, however now that the failure threshold is so low, it causes random unready pods once in a while due to the CPU hitting 100% or other transient problems.
My question: How can I make the pods terminate faster while keeping Readiness failure threshold a high number to reduce the downtime during pod restart? (Also, I would appreciate any other random tips that would make the pods restart faster (e.g. spend less time in ContainerCreating)
If you want to delete a Pod forcibly using kubectl version >= 1.5, do the following:
kubectl delete pods <pod> --grace-period=0 --forceIf you’re using any version of kubectl <= 1.4, you should omit the --force option and use:
kubectl delete pods <pod> --grace-period=0If even after these commands the pod is stuck on Unknown state, use the following command to remove the pod from the cluster:
kubectl patch pod <pod> -p '{"metadata":{"finalizers":null}}'Ref: https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/