We are using prometheus adapter to allow for custom metric auto scaling in k8s. We have everything setup where the HPA is getting the metric but it is exhibiting improper scale up even when the metric is below the target.

Polling the metric api server we can see the correct metric.

We can see the metric in prometheus over time where value never gets above 25% but scaling of the container still happens from 1-4.

We can see where the pod has a target value of 100 but has shown history of scaling from 1 to 4.

And the config for rule in prometheus adapter.

Pod Helm Config

Any help or direction to check next would be great. The expected result is we should not see any autoscaling as the target, 100, is always greater then the metric.

UPDATE 1: Ran on watch and got this

with this coming from prometheus -v 8