I'm using ingress-nginx as an Ingress controller for one of my services running over K8S (I'm using the nginx-0.20.0 release image with no specific metrics configurations in the K8S configmap the ingress-controller is using).
The nginx-ingress-controller pods are successfully scraped into my Prometheus server but all ingress metrics (e.g. nginx_ingress_controller_request_duration_seconds_bucket) show up with path="/" regardless of the real path of the handled request.
Worth noting that when I look at the ingress logs - the path is logged correctly.
How can I get the real path noted in the exported metrics?
Thanks!
The Path attribute in the NGINX metrics collected by prometheus derives from the Ingress definition yaml.
For example, if your ingress is:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: <some-k8s-ingress-name>
namespace: <some-k8s-namespace-name>
spec:
rules:
- host: <hostname>
http:
paths:
- backend:
serviceName: <some-k8s-service-name>
servicePort: <some-port>
path: /Then although NGINX will match any URL to your service, it'll all be logged under the path "/" (as seen here).
If you want metrics for a specific URL, you'll need to explicitly specify it like this (notice the ordering of rules):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: <some-k8s-ingress-name>
namespace: <some-k8s-namespace-name>
spec:
rules:
- host: <hostname>
http:
paths:
- backend:
serviceName: <some-k8s-service-name>
servicePort: <some-port>
path: /more/specific/path
- backend:
serviceName: <some-k8s-service-name>
servicePort: <some-port>
path: /If I am understand right, this label was removed as causing high cardinality: https://github.com/kubernetes/ingress-nginx/issues/2924#issuecomment-424516095
And this is more general issue: https://github.com/kubernetes/ingress-nginx/pull/2701
Have to say that I am pretty agree - think about it: if you allow the route to be the label value, it can be pretty easy to DDoS your Prometheus - just call your service with a lot of unexisting routes.
All those calls will be translated into the Label Values for all ingress metrics....