I try to implement a HPA based on phpfpm's active process count. I installed prometheus, the prometheus-adapter and the exporter as sidecar to my PHP-Application. At the prometheus' UI get the correct metrics.

Here is my prometheus-adapter-config:

  - seriesQuery: 'phpfpm_active_processes{kubernetes_namespace!="",kubernetes_pod_name!=""}'
    resources:
      overrides:
        kubernetes_namespace:
          resource: "namespace"
        kubernetes_pod_name:
          resource: "pod"
    name:
      matches: "phpfpm_active_processes"
      as: "phpfpm_avg_processes"
    metricsQuery: 'avg(avg_over_time(phpfpm_active_processes[30s])) by (kubernetes_pod_name,kubernetes_namespace))'

If i call the k8s-api, i see the metrics are available:

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "namespaces/phpfpm_avg_processes",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "pods/phpfpm_avg_processes",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

But the HPA fails with an error: unable to get metric phpfpm_avg_processes: unable to fetch metrics from custom metrics API: Internal error occurred: unable to fetch metrics

If i try to fetch the metrics via kubectl, i'll get the error:

$ kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/phpfpm_avg_processes' | jq
Error from server (InternalError): Internal error occurred: unable to fetch metrics

The logs of the prometheus-adapter includes following messages:

2020-11-03T08:27:07.7920803Z I1103 08:27:07.791793       1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/mynamespace/pods/%2A/phpfpm_avg_processes?labelSelector=myapp.service%3Dtestapp: (13.6381ms) 500

I also tried adding --kubelet-insecure-tls and --kubelet-preferred-address-types=InternalIP to the metrics-server with no luck.

Info: for testing i use docker-for-mac

Can anybody help? I don't know how to debug or where to look at?!