I have a deployment similar to this:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: <my-deployment>
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: <my-deployment>
  minReplicas: 1
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 133

I'd like to optimize the resource usage by making the downscale happen more quickly - right now I see that it keeps pods up despite the cpu utilization looking like Current/Target value-70%/133%.

Clarification - I want this to affect this deployment only, not the entire cluster.