I have followed the instructions at https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/ to install the Prometheus and Grafana to monitor the performance of the ingress-nginx controller.

However, the instructions to access the Prometheus dashboard did not work for me.

Indeed, according to the instructions we should be able to access the service through one of the IPs returned by the following command:

kubectl get nodes --selector=kubernetes.io/role!=master -o jsonpath={.items[*].status.addresses[?\(@.type==\"InternalIP\"\)].address}

However, I do not understand how it is possible, since the IPs are internal to the cluster. Even in their documentation:

How can it possibly work if the IP addresses 10.192.0.2 10.192.0.3 10.192.0.4 are internal to the cluster?

Here is what I have:

mark@D-LP0003:~$ kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
grafana                              NodePort       10.0.6.223     <none>          3000:32275/TCP               23m
ingress-nginx-controller             LoadBalancer   10.0.254.117   40.76.145.210   80:32542/TCP,443:31960/TCP   51m
ingress-nginx-controller-admission   ClusterIP      10.0.110.16    <none>          443/TCP                      51m
prometheus-server                    NodePort       10.0.208.16    <none>          9090:32053/TCP               48m
mark@D-LP0003:~$

No external IP. OK, the documentation has it in mind already and suggests to run another command:

mark@D-LP0003:~$ kubectl get nodes -o wide
NAME                                STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-nodepool1-30294001-vmss000000   Ready    agent   14h   v1.17.9   10.240.0.4    <none>        Ubuntu 16.04.7 LTS   4.15.0-1092-azure   docker://3.0.10+azure
aks-nodepool1-30294001-vmss000001   Ready    agent   14h   v1.17.9   10.240.0.5    <none>        Ubuntu 16.04.7 LTS   4.15.0-1092-azure   docker://3.0.10+azure
aks-nodepool1-30294001-vmss000003   Ready    agent   11h   v1.17.9   10.240.0.6    <none>        Ubuntu 16.04.7 LTS   4.15.0-1092-azure   docker://3.0.10+azure
mark@D-LP0003:~$

Still no external IP. And at this point the documentation suggests to run yet another command:

mark@D-LP0003:~$ kubectl get nodes --selector=kubernetes.io/role!=master -o jsonpath={.items[*].status.addresses[?\(@.type==\"InternalIP\"\)].address}
10.240.0.4 10.240.0.5 10.240.0.6mark@D-LP0003:~$

I get back the same internal IPs.

The only way I found to access the dashboard is through the client side port forwarding:

kubectl port-forward svc/prometheus-server 9090:9090 -n ingress-nginx  &

And then I can access the server at http://localhost:9090.

But I do not want it. The service is supposed to be externally accessible, so how do I access it?