I'm attempting to auto-scale a Kubernetes deployment with an HPA using Prometheus custom metrics with the Prometheus Adapter. These custom metrics are published to Prometheus via another deployment in another namespace which every minute queries a REST API for a particular metric and then publish the value of that metric to Prometheus. From there the adapter should be able to query Prometheus for said metric, with some additional labels as query criteria, and publish that metric with a new name. From there the HPA should be able to pick up this metric and scale based on its value.

Here the labels for my deployment which the adapter supposedly bases its matching off of:

Labels:             app.kubernetes.io/instance=event-subscription-dev-dev
                    app.kubernetes.io/managed-by=Tiller-dev
                    app.kubernetes.io/name=event-subscription-dev
                    deployment-name=event-subscription-webhook-worker-dev
                    helm.sh/chart=event-subscription-0.1.0-dev

Here are the Prometheus Adapter Helm chart values/adapter rules:

logLevel: 1
metricsRelistInterval: 5s
prometheus:
  url: 'http://<prometheus-url>'
rules:
  custom:
    - seriesQuery: '{__name__="event_subscription_current_message_lag"}'
      name: 
        matches: "(.*)"
        as: '${1}_webhooks'
      resources: 
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      metricsQuery: 'sum(event_subscription_current_message_lag{queue="webhooks", container_name!="POD"})'
    - seriesQuery: '{__name__="event_subscription_current_message_lag"}'
      name: 
        matches: "(.*)"
        as: '${1}_webhook_retries'
      resources: 
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      metricsQuery: 'sum(event_subscription_current_message_lag{queue="webhook_retries", container_name!="POD"})'

And here is the metrics piece of my HPA spec:

 metrics:
  - type: Pods
    pods:
      metric:
        name: event_subscription_current_message_lag_webhooks
      target:
        type: AverageValue
        averageValue: 10
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 85

The problem I'm having here is not with the adapter querying for the metric and then publishing a new metric, but rather that the new metric has no value associated with it as the original metric does.

For example if I run kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 I do see my event_subscription_current_message_lag_webhooks and event_subscription_current_message_lag_webhook_retries metrics, but they don't have any value like the original event_subscription_current_message_lag metric does.

Here's output from kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/event-subscription/pods/*/event_subscription_current_message_lag"

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/event-subscription/pods/%2A/event_subscription_current_message_lag"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "event-subscription",
        "name": "activemq-message-lag-retrieval-7bfc46b948-jr8kp",
        "apiVersion": "/v1"
      },
      "metricName": "event_subscription_current_message_lag",
      "timestamp": "2019-11-08T22:09:53Z",
      "value": "1"
    }
  ]
}

And here's the output for event_subscription_current_message_lag_webhooks and event_subscription_current_message_lag_webhook_retries:

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/event-subscription/pods/%2A/event_subscription_current_message_lag_webhooks"
  },
  "items": []
}

...

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/event-subscription/pods/%2A/event_subscription_current_message_lag_webhook_retries"
  },
  "items": []
}

I'm confused as to how the adapter is able to, seemingly, find my original metric, query for it, publish the new metric, but without the value, I would expect which in this case is 1.