I'm trying to setup custom metrics based autoscaler on a EKS cluster (v1.13.10-eks-5ac0f1) but looks like the labelSelector filter for external metrics labels is not filtering.

Using k8s-prometheus-adapter and metrics-server(v0.3.6) I've managed to export metrics from prometheus as kubernetes external metrics.

The metric is correctly exported and visible on the kubernetes api:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/*/sqs_queue_messages"

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/external.metrics.k8s.io/v1beta1/namespaces/%2A/sqs_queue_messages"
  },
  "items": [
    {
      "metricName": "sqs_queue_messages",
      "metricLabels": {
        "__name__": "sqs_queue_messages",
        ...
        "queue_name": "temp-queue"
      },
      "timestamp": "2019-11-07T21:14:44Z",
      "value": "0"
    },
    {
      "metricName": "sqs_queue_messages",
      "metricLabels": {
        "__name__": "sqs_queue_messages",
        ...
        "queue_name": "random-queue"
      },
      "timestamp": "2019-11-07T21:14:44Z",
      "value": "0"
    }
  ]
}

horizontal-pod-autoscaler.yml

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - external:
        metricName: sqs_queue_messages
        metricSelector:
          matchLabels:
            queue_name: temp-queue
        targetAverageValue: "100"
      type: External

The problem is that the HPA is not selecting just the metric with the matched label, infact by looking at the logs I can see that the following call is performed

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/*/sqs_queue_messages?labelSelector=queue_name%3Dtemp-queue"

The expected result is just 1 item (the one matching queue_name: temp-queue label) but instead the filter is ignored and all the results are returned.