I recently deployed a testing version of my app in a different namespace in the same GKE cluster. My application makes use of GPU resources and my HPA is supposed to auto-scale based on GPU load.

I noticed that the HPA was scaling to the maximum amount of replicas despite having zero load.

The reason seems to be that the kubernetes.io|container|accelerator|duty_cycle metric is not specific to a namespace. And my matchLabels filter targets the api deployment, which exists in two namespaces (default and staging).

How can I additionally filter by namespace? I tried metadata.namespace but that does not work.

Here is the HPA:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
  namespace: staging
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: External
      external:
        metricName: kubernetes.io|container|accelerator|duty_cycle
        metricSelector:
          matchLabels:
            metadata.system_labels.top_level_controller_name: api
            metadata.system_labels.top_level_controller_type: Deployment
        targetAverageValue: "75"a

$ kubectl describe hpa api-hpa
Name:                                                                       api-hpa
Namespace:                                                                  staging
Labels:                                                                     <none>
Annotations:                                                                <none>
CreationTimestamp:                                                          Sun, 27 Jun 2021 23:53:33 +0200
Reference:                                                                  Deployment/api
Metrics:                                                                    ( current / target )
  "kubernetes.io|container|accelerator|duty_cycle" (target average value):  65 / 75
Min replicas:                                                               1
Max replicas:                                                               3
Deployment pods:                                                            3 current / 3 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric kubernetes.io|container|accelerator|duty_cycle(&LabelSelector{MatchLabels:map[string]string{metadata.system_labels.top_level_controller_name:api,metadata.system_labels.top_level_controller_type: Deployment,},MatchExpressions:[]LabelSelectorRequirement{},})
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range