We have a GKE cluster (1.11) and implemented HPA based on memory utilization for pods. During our testing activity, we have observed HPA behavior is not consistent, HPA is not scaling pods even though the target value is met. We have also noticed that, HPA events is not giving us any response data ( either scaling or downscaling related info).

Example

kubectl get hpa

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE

com-manh-cp-organization Deployment/com-manh-cp-organization 95%/90% 1 25 1 1d

kubectl describe hpa com-manh-cp-organization

Name:                                                     com-manh-cp-organization
Namespace:                                                default
Labels:                                                   app=com-manh-cp-organization
                                                          stereotype=REST
Annotations:                                              kubectl.kubernetes.io/last-applied-configuration:
                                                            {"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"labels":{"app":"com-manh-cp-organizatio...
CreationTimestamp:                                        Tue, 12 Feb 2019 18:02:12 +0530
Reference:                                                Deployment/com-manh-cp-organization
Metrics:                                                  ( current / target )
  resource memory on pods  (as a percentage of request):  95% (4122087424) / 90%
Min replicas:                                             1
Max replicas:                                             25
Deployment pods:                                          1 current / 1 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           <none>

• Cluster version : 1.11.6
• Cloud service : GKE
• Metric : memory
• Target : targetAverageUtilization

Any inputs will be much appreciated and let us know if we can debug HPA implementation.

Thanks.