Minikube Nginx Ingress not finding service endpoint

8/26/2021

I'm having some trouble getting the Nginx ingress controller working in my Minikube cluster. It's likely to be some faults in Ingress configuration but I cannot pick it out.

First, I deployed a service and it worked well without ingress.

kind: Service
apiVersion: v1
metadata:
  name: online
  labels:
    app: online
spec:
  selector:
    app: online
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 5001
  type: LoadBalancer

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: online
  labels:
    app: online
spec:
  replicas: 1
  selector:
    matchLabels:
      app: online
  template:
    metadata:
      labels:
        app: online
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "online"
        dapr.io/app-port: "5001"
        dapr.io/log-level: "debug"
        dapr.io/sidecar-liveness-probe-threshold: "300"
        dapr.io/sidecar-readiness-probe-threshold: "300"
    spec:
      containers:
      - name: online
        image: online:latest
        ports:
        - containerPort: 5001
        env:
        - name: ADDRESS
          value: ":5001"
        - name: DAPR_HTTP_PORT
          value: "8080"
        imagePullPolicy: Never

Then check its url

minikube service online --url
http://192.168.49.2:32323

It looks ok for requests.

curl http://192.168.49.2:32323/userOnline
OK

After that I tried to apply nginx ingress offered by minikube. I installed ingress and run an example by referring to this and it's all ok.

Lastly, I configured my Ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: online-ingress
  annotations:
spec:
  rules:
    - host: online
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: online
                port:
                  number: 8080

And changed /etc/hosts by adding line

192.168.49.2    online

And Test:

curl online/userOnline
502 Bad Gateway

The logs are like this:

192.168.49.1 - - [26/Aug/2021:09:45:56 +0000] "GET /userOnline HTTP/1.1" 502 150 "-" "curl/7.68.0" 80 0.002 [default-online-8080] [] 172.17.0.5:5001, 172.17.0.5:5001, 172.17.0.5:5001 0, 0, 0 0.004, 0.000, 0.000 502, 502, 502 578ea1b1471ac973a2ac45ec4c35d927
2021/08/26 09:45:56 [error] 2514#2514: *426717 upstream prematurely closed connection while reading response header from upstream, client: 192.168.49.1, server: online, request: "GET /userOnline HTTP/1.1", upstream: "http://172.17.0.5:5001/userOnline", host: "online"
2021/08/26 09:45:56 [error] 2514#2514: *426717 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.49.1, server: online, request: "GET /userOnline HTTP/1.1", upstream: "http://172.17.0.5:5001/userOnline", host: "online"
2021/08/26 09:45:56 [error] 2514#2514: *426717 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.49.1, server: online, request: "GET /userOnline HTTP/1.1", upstream: "http://172.17.0.5:5001/userOnline", host: "online"
W0826 09:45:56.918446       7 controller.go:977] Service "default/online" does not have any active Endpoint.
I0826 09:46:21.345177       7 status.go:281] "updating Ingress status" namespace="default" ingress="online-ingress" currentValue=[] newValue=[{IP:192.168.49.2 Hostname: Ports:[]}]
I0826 09:46:21.349078       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"online-ingress", UID:"b69e2976-09e9-4cfc-a8e8-7acb51799d6d", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"23100", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

I found the error is very about annotations of Ingress. If I changed it to:

  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1

The error would be:

404 page not found

and logs:

I0826 09:59:21.342251       7 status.go:281] "updating Ingress status" namespace="default" ingress="online-ingress" currentValue=[] newValue=[{IP:192.168.49.2 Hostname: Ports:[]}]
I0826 09:59:21.347860       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"online-ingress", UID:"8ba6fe97-315d-4f00-82a6-17132095fab4", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"23760", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
192.168.49.1 - - [26/Aug/2021:09:59:32 +0000] "GET /userOnline HTTP/1.1" 404 19 "-" "curl/7.68.0" 80 0.002 [default-online-8080] [] 172.17.0.5:5001 19 0.000 404 856ddd3224bbe2bde9d7144b857168e0

Other infos.

NAME     TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
online   LoadBalancer   10.111.34.87   <pending>     8080:32323/TCP   6h54m

The example I mentioned above is a NodePort service and mine is a LoadBalancer, that's the biggest difference. But I don't know why it does not work for me.

-- tanxin
kubernetes
kubernetes-ingress
minikube
nginx

1 Answer

8/30/2021

Moving this out of comments so it will be visible.


Ingress

Main issue was with path in ingress rule since application serves traffic on online/userOnline. If requests go to online then ingress returns 404.

Rewrite annotation is not needed in this case as well.

ingress.yaml should look like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: online-ingress
#  annotations:
spec:
  rules:
    - host: online
      http:
        paths:
          - path: /userOnline
            pathType: Prefix
            backend:
              service:
                name: online
                port:
                  number: 8080

More details about ingress


LoadBalancer on Minikube

Since minikube is considered as bare metal installation, to get external IP for service/ingress, it's necessary to use specially designed metallb solution.

MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.

It ships as add-on for minikube and can be enabled with:

minikube addons enable metallb

And it needs to create a configMap with setup. Please refer to metallb configuration

-- moonkotte
Source: StackOverflow