I was following this walk through (partially, am using EKS. https://itnext.io/horizontal-pod-autoscale-with-custom-metrics-8cb13e9d475
I manage to get scaled up one deployment with this http_requests_total metric.
Now, I am trying to add a new metric. I have prometheus server, and it already scrapes cloudwatch and I have aws_sqs_approximate_age_of_oldest_message_maximum value there for many of my queues.
In the similar manner to the mentioned tutorial, I am adding the definition of a metric:
- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "^(.*)_total"
as: "${1}_per_second"
metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'vs
- seriesQuery: 'aws_sqs_approximate_age_of_oldest_message_maximum{queue_name!=""}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
metricsQuery: '<<.Series>>{<<.LabelMatchers>>}'Or some version of the bottom one. But, I can never see it in: kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq
No matter what I try.
Any ideas how to move forward? Thanks!
If you don't see the metric in /apis/custom.metrics.k8s.io/v1beta1 it means that the Prometheus Adapter couldn't discover it.
The Prometheus Adapter discovers metrics by using the value of your seriesQuery field for an /api/v1/series request to Prometheus (done periodically with a frequency defined by the relist interval).
Things to try:
What do you get if you make the following request to Prometheus?
http://<prometheus-ip>:9090/api/v1/series? match[]=aws_sqs_approximate_age_of_oldest_message_maximum{queue_name!=""}&start=<current-timestamp-sec>What do you get if you drop the following in the query text box of the Prometheus UI and press Execute?
aws_sqs_approximate_age_of_oldest_message_maximum{queue_name!=""}If you get no data back in either case, then you just don't have any time series in Prometheus that match your seriesQuery specification.