I have a service (installed as Helm subchart of another app) that is correctly reachable on a Minikube environment, but that is not reachable when deployed on a remote Kubernetes cluster.

The Service should be reachable via an ingress.

Ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: helper-service
  labels:
    helm.sh/chart: "helper-service-0.2.3"
    app.kubernetes.io/name: "helper-service"
    app.kubernetes.io/version: "2.1"
    app.kubernetes.io/managed-by: "Helm"
  annotations:
    # These annotations don't seem to make any difference
    # traefik.ingress.kubernetes.io/ingress.class: traefik
    # traefik.ingress.kubernetes.io/ssl-proxy-headers: X-Forwarded-Proto:https
    # traefik.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
    - host: "helper-service.example.com"
      http:
        paths:
          - path: /
            backend:
              serviceName: helper-service
              servicePort: service-port

Service

apiVersion: v1
kind: Service
metadata:
  name: helper-service
  labels:
    helm.sh/chart: "helper-service-1.0.0"
    app.kubernetes.io/name: "helper-service"
    app.kubernetes.io/version: "2.13.1"
    app.kubernetes.io/managed-by: "Helm"
spec:
  selector:
    app.kubernetes.io/name: "helper-service"
  type: ClusterIP
  ports:
    - port: 8080
      name: service-port
      targetPort: 8080
      protocol: TCP

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helper-service
  labels:
    helm.sh/chart: "helper-service-1.0.0"
    app.kubernetes.io/name: "helper-service"
    app.kubernetes.io/version: "2.1"
    app.kubernetes.io/managed-by: "Helm"
spec:
  replicas: 1
  revisionHistoryLimit: 3
  strategy:
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: "helper-service"
  template:
    metadata:
      annotations: {}
      labels:
        app.kubernetes.io/name: "helper-service"
    spec:
      containers:
      - name: helper-service
        image: "myregistry/myimage:myversion"
        env:
        - name: HELPER-SERVICE_BIND
          value: ":8080"
        ports:
        - containerPort: 8080

Sending requests to helper-service.example.com always return a 404 not found error from Nginx. Given the response, seems like Nginx is reachable but isn't able to provide access to the service specified in the Ingress.

However, the parent-chart Helm app, on the same cluster/namespace, is reachable via Ingress on a NodePort service and has no problems in being accessed.

Both services are running correctly:

$ k get all
NAME                                         READY   STATUS             RESTARTS   AGE
pod/myapp-645cd66d9-g84vm                    2/2     Running            0          51m
pod/myapp-helper-service-fbc6bb4d8-zklf2     1/1     Running            0          4d19h

NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
service/myapp                   NodePort    10.10.10.11    <none>        8080:31033/TCP      110d
service/myapp-helper-service    ClusterIP   10.10.10.61    <none>        8080/TCP            110d

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/myapp                   1/1     1            1           110d
deployment.apps/myapp-helper-service    1/1     1            1           110d

and the ingresses looks to be there correctly:

$ k get ingresses
NAME                    HOSTS                                             ADDRESS       PORTS     AGE
myapp                   myapp.example.com                                 37.44.1.172   80, 443   116d
myapp-helper-service    myapp-helper-service.example.com                                80        116d

with the ingress details being:

$ k describe ingress/myapp-helper-service
Name:             myapp-helper-service
Namespace:        my-namespace
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" is forbidden: User cannot get resource "endpoints" in API group "" in the namespace "kube-system">)
Rules:
  Host                                             Path  Backends
  ----                                             ----  --------
  myapp-helper-service.example.com
                                                   /   myapp-helper-service:service-port (172.25.0.207:8080)
                                                   Events:                                            <none>

and the deployed service being:

$ k describe service/myapp-helper-service
Name:              myapp-helper-service
Namespace:         mynamespace
Labels:            app.kubernetes.io/instance=myapp
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=helper-service
                   app.kubernetes.io/version=2.1
                   helm.sh/chart=helper-service-0.2.3
Annotations:       meta.helm.sh/release-name: myapp
                   meta.helm.sh/release-namespace: mynamespace
Selector:          app.kubernetes.io/instance=myapp,app.kubernetes.io/name=helper-service
Type:              ClusterIP
IP:                10.10.10.61
Port:              service-port  8080/TCP
TargetPort:        8080/TCP
Endpoints:         172.25.0.207:8080
Port:              metrics-port  8081/TCP
TargetPort:        8081/TCP
Endpoints:         172.25.0.207:8081
Session Affinity:  None
Events:            <none>

I tried to enable/disable the add Traefik annotations to see if anything changed, but the result is the same (still 404 from Nginx 🤷‍♂️).

1. Why the same configuration works correctly on Minikube?
2. What does the 404 http status suggest here? (ingress reachable but service not found?)
3. How do I debug the reachability from inside the cluster?
4. How do I debug the reachability from outside the cluster/ingress?
5. If the setup is correct here, what else should I check?