Kubernetes Horisontal Pod Autoscaling (HPA) modifies my custom metric: StackDriver displays correct metric, but HPA shows another number. For example, StackDrives value is 118K, but HPA displays 1656144. I understand that HPA use some conversation for floating numbers, but my metric is integer: Unit: number Kind: Gauge Value type: Int64.
Running in GKE 1.11.7.
Any ideas?
In your HPA manifest you did not specify value of --horizontal-pod-autoscaler-sync-period flag. As default it is set to 15 seconds.
In your case it means that HPA value is amout of whole deployment queue in last 15 seconds. More information can be found in HPA Documentation.
As you mentioned in StackDriver you used GAUGE metric which measures a value at a particular point in time - Stackdriver
In short, StackDriver shows current value in the exact time, HPA values is amount of last 15 seconds.
if you specify targetValue it will be a whole number, so there won't be scaling down of pods. If you use targetAverageValue it will calculate based on the number of pods created.