horizontalpodautoscaler min and max replica are 1 creates 2 pods

4/20/2019

I am trying to run horizontalpodautoscaler resource defining mix and max replica as 1. somehow once it started to run 2 instances of the pod are running in the same time then 1 pod is terminated. End state I have 1 pod running.

Is this the normal behaviour of HPA resource that somehow its created replicaset with 2 pods even the maximum is one

Thanks

-- George Armand
kubernetes

2 Answers

4/20/2019

Generally it will not happen. The only reason it could occur if hpa target metric target is very low and starting the application is crossing the target CPU threshold causing scaling of pods.

You can check the HPA status, events section can explain the reason for scaling.

HPA status: kubectl describe HPA_NAME

Check below example for reference

    Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  40% (406m) / 50%
Min replicas:                                          10
Max replicas:                                          100
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
  ScalingActive   True    ValidMetricFound    the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired replica count is within the acceptible range
Events:
  Type    Reason             Age                   From                       Message
  ----    ------             ----                  ----                       -------
  Normal  SuccessfulRescale  35m (x1216 over 12d)  horizontal-pod-autoscaler  New size: 10; reason: All metrics below target
-- Shashank Sinha
Source: StackOverflow

4/21/2019

These are the events flow from all relevant resources:

HPA Events:
NAME            REFERENCE                  TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
example-ms   Deployment/example-ms   <unknown>/50%   1         1         0          0s
example-ms   Deployment/example-ms   <unknown>/50%   1         1         0         1s
example-ms   Deployment/example-ms   <unknown>/50%   1         1         2         3s
example-ms   Deployment/example-ms   <unknown>/50%   1         1         1         33s


Replica Set Events:
NAME                              DESIRED   CURRENT   READY     AGE
example-ms-59c9b45565   1         1         0         1s
example-ms-76896c7f7f   1         1         1         2s
example-ms-59c9b45565   1         1         1         2s
example-ms-76896c7f7f   1         1         1         32s
example-ms-59c9b45565   1         1         1         32s
example-ms-76896c7f7f   0         1         1         32s
example-ms-76896c7f7f   0         1         1         32s
example-ms-76896c7f7f   0         0         0         32s

Pod Events:
NAME                                    READY     STATUS              RESTARTS   AGE
example-ms-59c9b45565-6b5v8   0/1       ContainerCreating   0         0s
example-ms-76896c7f7f-cth7x   1/1       Running   0         2s
example-ms-59c9b45565-6b5v8   1/1       Running   0         2s
example-ms-59c9b45565-d2k8w   0/1       Terminating   0         2m
example-ms-59c9b45565-d2k8w   0/1       Terminating   0         2m
example-ms-59c9b45565-d2k8w   0/1       Terminating   0         2m
example-ms-59c9b45565-d2k8w   0/1       Terminating   0         2m
example-ms-76896c7f7f-cth7x   1/1       Terminating   0         32s
example-ms-76896c7f7f-cth7x   0/1       Terminating   0         1m
example-ms-76896c7f7f-cth7x   0/1       Terminating   0         1m
example-ms-76896c7f7f-cth7x   0/1       Terminating   0         1m




Deployment Spec:
spec:
  minReadySeconds: 30
  progressDeadlineSeconds: 120
  replicas: 1
  revisionHistoryLimit: 15
  selector:



Deployment status:
tatus:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2019-04-21T08:51:03Z
    lastUpdateTime: 2019-04-21T08:51:35Z
    message: ReplicaSet "example-ms-59c9b45565" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: 2019-04-21T16:53:02Z
    lastUpdateTime: 2019-04-21T16:53:02Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
-- George Armand
Source: StackOverflow