I have configured Prometheus-adapter to fetch custom metrics from Prometheus. When I execute the command: kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
Following is the result.
{
"name": "namespaces/envoy_http_ingress_http_downstream_cx_http1",
"singularName": "",
"namespaced": false,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "namespaces/envoy_cluster_xds_cluster_upstream_cx_rx_bytes",
"singularName": "",
"namespaced": false,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "jobs.batch/statsd_exporter_lines",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "pods/fs_writes_merged",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
My HPA configuration is as follows:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: scale
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: billing-app
minReplicas: 1
maxReplicas: 10
# targetCPUUtilizationPercentage: 50
metrics:
- type: External
external:
metricName: fs_writes_merged
targetValue: 100
Hpa results in unknown. Not sure why it is not able to fetch metrics.
Hpa must be able to read the custom metrics.
Since your HPA configuration declares the metric as type: External
, the HPA tries to fetch it from the External Metrics API (/apis/custom.metrics.k8s.io
), but the Prometheus Adapter exposes it on the Custom metrics API (/apis/custom.metrics.k8s.io
).
Since your metric comes from the Pods of the Deployment that you're trying to autoscale, you should use the Pods
metric type. So, change your HPA configuration to:
# ...
metrics:
- type: Pods
pods:
metricName: fs_writes_merged
targetValue: 100
You can see all the available HPA metric types and their usages here:
kubectl explain --api-version=autoscaling/v2beta1 hpa.spec.metrics
There are four metric types, and they map to the various metric APIs as follows:
Resource
: Resource Metrics API (/apis/metrics.k8s.io/v1beta1
)Pods
: Custom Metrics API (/apis/custom.metrics.k8s.io/v1beta1
)Object
: Custom Metrics API (/apis/custom.metrics.k8s.io/v1beta1
)External
: External Metrics API (/apis/external.metrics.k8s.io/v1beta1
)See HPA docs, and design documents for Resource Metrics API, Custom Metrics API, and External Metrics API.