I've prometheus/Grafana installed on GKE, and i'm trying expose Prometheus using Ingress ( ref link - https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress https://cloud.google.com/community/tutorials/nginx-ingress-gke

Here is the Prometheus install (in namespace - monitoring) :

Karans-MacBook-Pro:ingress-ns karanalang$ kc get all -n monitoring
NAME                                       READY   STATUS       RESTARTS   AGE
pod/grafana-5ccfb68647-xj577               1/1     Running      0          165m
pod/prometheus-operator-85d84bb848-qs9f8   1/1     Running      0          165m
pod/prometheus-prometheus-0                2/2     Running      0          165m
pod/prometheus-prometheus-1                2/2     Running      0          165m
pod/prometheus-prometheus-2                2/2     Running      0          165m

NAME                          TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)          AGE
service/grafana               ClusterIP      10.80.9.14    <none>           3000/TCP         6d2h
service/prometheus-operated   ClusterIP      None          <none>           9090/TCP         6d2h
service/prometheus-operator   ClusterIP      None          <none>           8080/TCP         6d2h

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana               1/1     1            1           6d2h
deployment.apps/prometheus-operator   1/1     1            1           6d2h

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-5ccfb68647               1         1         1       6d2h
replicaset.apps/prometheus-operator-85d84bb848   1         1         1       6d2h

NAME                                     READY   AGE
statefulset.apps/prometheus-prometheus   3/3     6d2h

I've installed NGINX Controller using helm charts in namespace -> ingress-helm

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install nginx-ingress ingress-nginx/ingress-nginx -n ingress-helm

Karans-MacBook-Pro:ingress-ns karanalang$ kc get deploy,svc -n ingress-helm -o wide
NAME                                                     READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                                                                                               SELECTOR
deployment.apps/nginx-ingress-ingress-nginx-controller   1/1     1            1           117m   controller   k8s.gcr.io/ingress-nginx/controller:v1.1.1@sha256:0bc88eb15f9e7f84e8e56c14fa5735aaa488b840983f87bd79b1054190e660de   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx

NAME                                                       TYPE           CLUSTER-IP    EXTERNAL-IP       PORT(S)                      AGE    SELECTOR
service/nginx-ingress-ingress-nginx-controller             LoadBalancer   10.80.5.117   104.196.128.221   80:30894/TCP,443:32354/TCP   117m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
service/nginx-ingress-ingress-nginx-controller-admission   ClusterIP      10.80.1.68    <none>            443/TCP                      117m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx

Ingress resources is created in namespace - monitoring, which also has Prometheus/Grafana installed Yaml to create the Ingress is below, also the

Karans-MacBook-Pro:ingress-ns karanalang$ cat Ingress-resource-jan12-withoutHost-helm.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
  name: ingress-monitoring-helm
  namespace: ingress-helm
spec:
  rules:
  - host: "104.196.128.221.nip.io"	
    http:
      paths:
      - backend:
          service:
            name: prometheus-operated
            port:
              number: 9090
        path: /*
        pathType: ImplementationSpecific


Karans-MacBook-Pro:ingress-ns karanalang$ kc get ingress ingress-monitoring-helm -n monitoring
NAME                      CLASS    HOSTS                    ADDRESS           PORTS   AGE
ingress-monitoring-helm   <none>   104.196.128.221.nip.io   104.196.128.221   80      83s

when i do a curl on the host(or the External IP), here is what i get :

Karans-MacBook-Pro:ingress-ns karanalang$ curl 104.196.128.221.nip.io
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

what needs to be done to fix this issue with the Ingress ? With the ingress created, i should be able to see the Prometheus UI - however that is not happening.

Pls note : Prometheus & Grafana are up and running, and i'm able to access then using LoadBalancer as well as NodePort services. Also - if i just create an Ingress (kubernetes.io/ingress.class: "gce"), and curl the IP it takes me to Prometheus yaml file with -> kubernetes.io/ingress.class: "gce"

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "gce"
  name: ingress-monitoring-withhost
  namespace: monitoring
spec:
  rules:
  - host: 34.120.109.103.nip.io
    http:
      paths:
      - backend:
          service:
            name: prometheus-operated
            port:
              number: 9090
        path: /*
        pathType: ImplementationSpecific

tia !

UPDATE 01/13 :

Based on comment from @Sam Stoelinga, i updated the ingress to use path /metrics, that seems to be giving the metrics On the other hand, when i use ingress.class=gce, it gives me the Prometheus UI. Any ideas why there is a difference here ?

Karans-MacBook-Pro:prometheus-yamls karanalang$ kc describe ingress ingress-monitoring-helm -n monitoring
Name:             ingress-monitoring-helm
Namespace:        monitoring
Address:          104.196.128.221
Default backend:  default-http-backend:80 (10.76.0.8:8080)
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  104.196.128.221.nip.io  
                          /metrics   prometheus-operated:9090 (10.76.0.3:9090,10.76.0.4:9090,10.76.2.3:9090)
                          /grafana   grafana:3000 (10.76.0.5:3000)
Annotations:              kubernetes.io/ingress.class: nginx
                          nginx.ingress.kubernetes.io/ssl-redirect: false
Events:                   <none>