I have a problem with Kubernetes and Prometheus' custom-metrics on Amazon's AWS. By default, CPU and memory metrics are working just fine. Prometheus http_requests isn't, this is the error:

$ kubectl describe hpa hpa-deploy
Name:                       hpa-deploy
Namespace:                  default
Labels:                     <none>
Annotations:                kubectl.kubernetes.io/last-applied-configuration:
                              {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-deploy","namespace":"default...
CreationTimestamp:          Thu, 06 Jun 2019 11:06:48 +0000
Reference:                  Deployment/django
Metrics:                    ( current / target )
  "http_requests" on pods:  <unknown> / 2k
Min replicas:               1
Max replicas:               10
Deployment pods:            1 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server could not find the metric http_requests for pods
Events:
  Type     Reason               Age                     From                       Message
  ----     ------               ----                    ----                       -------
  Warning  FailedGetPodsMetric  8m53s (x414 over 114m)  horizontal-pod-autoscaler  unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
  Warning  FailedGetPodsMetric  3m48s (x12 over 6m36s)  horizontal-pod-autoscaler  unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server could not find the metric http_requests for pods

I installed Prometheus using helm as sugested by the github project and checked the api:

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": []
}

Then added the following rule:

$ kubectl edit cm my-release-prometheus-adapter
    rules:
    - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
      resources:
        overrides:
          kubernetes_namespace: {resource: "namespace"}
          kubernetes_pod_name: {resource: "pod"}
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

The walkthrough said that the return of the api check should have values inside the "resources":[] after the new rule was added, but there is none and I don't know why.

This is my hpa code:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-deploy
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: django
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests
        target:
          type: Value
          averageValue: 2k

Also, I'm using an Nginx based Ingress Controller, but ingress and service's hpa kubectl describe shows that:

$ kubectl describe hpa hpa-ingress
Name:                                                      hpa-ingress
Namespace:                                                 default
Labels:                                                    <none>
Annotations:                                               kubectl.kubernetes.io/last-applied-configuration:
                                                             {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-ingress","namespace":"defaul...
CreationTimestamp:                                         Thu, 06 Jun 2019 11:06:48 +0000
Reference:                                                 Ingress/test-ingress
Metrics:                                                   ( current / target )
  "http_requests" on Ingress/test-ingress (target value):  <unknown> / 2k
Min replicas:                                              1
Max replicas:                                              10
Ingress pods:                                              0 current / 0 desired
Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable to get the target's current scale: the server could not find the requested resource
Events:
  Type     Reason          Age                     From                       Message
  ----     ------          ----                    ----                       -------
  Warning  FailedGetScale  2m40s (x473 over 122m)  horizontal-pod-autoscaler  the server could not find the requested resource

I don't know for sure if I have to export manually http_requests metrics for Pods and, if this is the case, how do I do it? The documentation is all "copy and paste and everything will work just fine", but it isn't. Please, if possible, the more details the better, I'm really new at this subject. Thanks a lot.