I am trying to get a new cluster setup on Google Kubernetes Engine AKA GKE. I am using helm and added to the repo ingress-nginx https://kubernetes.github.io/ingress-nginx and grafana https://grafana.github.io/helm-charts

I was following this guide but I was not able to access an internal ip or get the ingress working to where I could see the dashboard. I have then tried this guide using their helm deployment for Loki Stack (Loki, Promtail, Grafana, Prometheus) with persistent volume claim.

That combined with opening port 8443 up and I was still getting a 503 error. I made this ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - http:
        paths:
          - path: /pro
            pathType: Prefix
            backend:
              service:
                name: loki-prometheus-server
                port:
                  number: 9090
          - path: /graf
            pathType: Prefix
            backend:
              service:
                name: loki-grafana
                port:
                  number: 3000

I tried many different ports including 80 but it was a no go.

loki-prometheus-server service:

Name:              loki-prometheus-server
Namespace:         ingress-nginx
Labels:            app=prometheus
                   app.kubernetes.io/managed-by=Helm
                   chart=prometheus-11.16.9
                   component=server
                   heritage=Helm
                   release=loki
Annotations:       cloud.google.com/neg: {"ingress":true}
                   meta.helm.sh/release-name: loki
                   meta.helm.sh/release-namespace: ingress-nginx
Selector:          app=prometheus,component=server,release=loki
Type:              ClusterIP
IP Families:       <none>
IP:                10.***
IPs:               10.***
Port:              http  80/TCP
TargetPort:        9090/TCP
Endpoints:         10.***:9090
Session Affinity:  None
Events:            <none>

loki-grafana service:

Name:              loki-grafana
Namespace:         ingress-nginx
Labels:            app.kubernetes.io/instance=loki
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=grafana
                   app.kubernetes.io/version=7.5.0
                   helm.sh/chart=grafana-5.7.10
Annotations:       cloud.google.com/neg: {"ingress":true}
                   meta.helm.sh/release-name: loki
                   meta.helm.sh/release-namespace: ingress-nginx
Selector:          app.kubernetes.io/instance=loki,app.kubernetes.io/name=grafana
Type:              ClusterIP
IP Families:       <none>
IP:                10.***
IPs:               10.***
Port:              service  80/TCP
TargetPort:        3000/TCP
Endpoints:         10.***:3000
Session Affinity:  None
Events:            <none>

Step by Step:

Create new cluster on gcloud:

1. GKE Standard
2. default-pool->Nodes-> Machine type: e2-small
3. default-pool->Security-> Allow full access to all Cloud APIs selected Create
4. Create.

Connect to console or cloud shell

Then:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

kubectl create namespace ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx

Everything seems normal so far.

Go to VPC network-> Firewall -> Find the one with 80,443 in Protocols/ports Then add 8443 to it.

Install loki:

helm upgrade --install loki grafana/loki-stack --namespace ingress-nginx  --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=true,loki.persistence.storageClassName=standard,loki.persistence.size=5Gi

Everything still seems to be going fine but if I go to my external load balancer I get 404 not found / nginx

I add this ingress:

using kubectl apply -f ingress.yml -n ingress-nginx

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: "nginx"
    prometheus.io/scrape: "true"
spec:
  rules:
    - http:
        paths:
          - path: /pro
            pathType: Prefix
            backend:
              service:
                name: loki-prometheus-server
                port:
                  number: 80
          - path: /graf
            pathType: Prefix
            backend:
              service:
                name: loki-grafana
                port:
                  number: 80

And get a 404 page not found in text on /pro And get redirected to /login with a 404 not found nginx page at /graf

What am I doing wrong?