I have a problem with Kubernetes and Prometheus' custom-metrics on Amazon's AWS. By default, CPU and memory metrics are working just fine. Prometheus http_requests isn't, this is the error:
$ kubectl describe hpa hpa-deploy
Name: hpa-deploy
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-deploy","namespace":"default...
CreationTimestamp: Thu, 06 Jun 2019 11:06:48 +0000
Reference: Deployment/django
Metrics: ( current / target )
"http_requests" on pods: <unknown> / 2k
Min replicas: 1
Max replicas: 10
Deployment pods: 1 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetPodsMetric the HPA was unable to compute the replica count: unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server could not find the metric http_requests for pods
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetPodsMetric 8m53s (x414 over 114m) horizontal-pod-autoscaler unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
Warning FailedGetPodsMetric 3m48s (x12 over 6m36s) horizontal-pod-autoscaler unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server could not find the metric http_requests for pods
I installed Prometheus using helm as sugested by the github project and checked the api:
$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": []
}
Then added the following rule:
$ kubectl edit cm my-release-prometheus-adapter
rules:
- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "^(.*)_total"
as: "${1}_per_second"
metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
The walkthrough said that the return of the api check should have values inside the "resources":[] after the new rule was added, but there is none and I don't know why.
This is my hpa code:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: hpa-deploy
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: django
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: http_requests
target:
type: Value
averageValue: 2k
Also, I'm using an Nginx based Ingress Controller, but ingress and service's hpa kubectl describe shows that:
$ kubectl describe hpa hpa-ingress
Name: hpa-ingress
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-ingress","namespace":"defaul...
CreationTimestamp: Thu, 06 Jun 2019 11:06:48 +0000
Reference: Ingress/test-ingress
Metrics: ( current / target )
"http_requests" on Ingress/test-ingress (target value): <unknown> / 2k
Min replicas: 1
Max replicas: 10
Ingress pods: 0 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale False FailedGetScale the HPA controller was unable to get the target's current scale: the server could not find the requested resource
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetScale 2m40s (x473 over 122m) horizontal-pod-autoscaler the server could not find the requested resource
I don't know for sure if I have to export manually http_requests metrics for Pods and, if this is the case, how do I do it? The documentation is all "copy and paste and everything will work just fine", but it isn't. Please, if possible, the more details the better, I'm really new at this subject. Thanks a lot.