I have a Kubernetes cluster running in Google Kubernetes Engine where I wish to scale a deployment based on the number of messages a GCP Pub/Sub subscription has outstanding. I found and followed a guide from Google covering exactly this scenario which involves deploying the custom-metrics-stackdriver-adapter.
However, because my subscriptions name is over 63 characters in length when I run kubectl describe hpa <MY_HPA_NAME>
I get the following events.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetExternalMetric 12s (x2 over 27s) horizontal-pod-autoscaler invalid label value: "<MY_VERBOSE_SUBSCRIPTION_NAME>": must be no more than 63 characters
Warning FailedComputeMetricsReplicas 12s (x2 over 27s) horizontal-pod-autoscaler failed to get pubsub.googleapis.com|subscription|num_undelivered_messages external metric: invalid label value: "<MY_VERBOSE_SUBSCRIPTION_NAME>": must be no more than 63 characters
How do I go about using a Subscription name which is over 63 characters as a metric to trigger HPA?
I did try looking into what metric values where being exposed but when I queried Kubernetes Custom Metrics API I can't see any actual values for that metric by doing the following:
$ kubectl get --raw "http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/namespaces/*/pods/*/pubsub.googleapis.com|subscription|num_undelivered_messages" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/%2A/pods/%2A/pubsub.googleapis.com%7Csubscription%7Cnum_undelivered_messages"
},
"items": []
}
However, as you can see this returned an empty "items"
lists suggesting there were no values recorded. I assumed that this was due to all of my subscription names being over 63 characters in length. To test this I made a new subscription with a very short name and recreated the HPA to be triggered by this new subscription. This time the pods actually scaled how I expected but when I queried the Custom Metric API as above it still returned and empty "items"
list.
Should I expect to see values in that response (and their labels)? How was Kubernetes able to scale the pods if no "items"
were present?
Here is design documentation of the kubernetes. There is no way to avoid the restriction other than shortening the label name.
rfc1035/rfc1123 label (DNS_LABEL): An alphanumeric (a-z, and 0-9) string, with a maximum length of 63 characters, with the '-' character allowed anywhere except the first or last character, suitable for use as a hostname or segment in a domain name.
Check the External Metrics API instead of Custom Metrics API for metrics items.
$ kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq