What happened:

HPA does not scale up pods as expected after upgrade from kubernetes 1.10.6 to 1.12.8. In below snippet, although the scaling metric is beyond the target value, Replicas count is not increasing.

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
test-service-hpa Deployment/test-deploy 666m/600m 3 100 3 26d

test-deploy-5b985f9785-4znlb 1/2 Running 0 3h21m
test-deploy-5b985f9785-csld4 2/2 Running 0 10h
test-deploy-5b985f9785-tg52z 1/2 Running 0 3h18m

Its is because, HPA is not considering unready pods in desiredReplica calculation. In our use case, we mark pods as unready when the pod is handling traffic and we dont want any more traffic to be routed to this pod. I know the kubernets term unready pods is not very well used in our usecase. But just this behavior was working fine with kubernetes 1.10.6 and not in 1.12.8, would like to know if there is a workaround to fix it.

What you expected to happen:

HPA should consider unready pods as well while scaling up. Or need a flag to restore this behavior from 1.10.6.

How to reproduce it (as minimally and precisely as possible):

Create a simple deployment with minPod =3, maxPod=100.

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version): 1.12.8
OS (e.g: cat /etc/os-release): Ubuntu "18.04 LTS (Bionic Beaver)"
Kernel (e.g. uname -a): x86_64 x86_64 GNU/Linux