We have configured MetalLB since our K8s cluster is hosted on bare metal infrastructure. It seems to be running fine with all pods up and running.

[~]# kubectl get all -n metallb-system
NAME                             READY   STATUS    RESTARTS   AGE
pod/controller-b78574c59-47qfv   1/1     Running   0          24h
pod/speaker-4q2vm                1/1     Running   0          24h
pod/speaker-m8kwk                1/1     Running   0          24h
pod/speaker-t4rvs                1/1     Running   0          24h

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/speaker   3         3         3       3            3           kubernetes.io/os=linux   24h

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   1/1     1            1           24h

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-b78574c59   1         1         1       24h

We have configured ingress controller via helm from https://github.com/kubernetes/ingress-nginx/releases/tag/helm-chart-3.29.0 and updating hostNetwork,ingressClass,kind to true,ingress-nginx,DaemonSet respectively in file values.yaml. The helm installation seems to have worked fine with all daemonset pods running and an LB ip provided to created ingress controller service.

[~]# kubectl get all -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
pod/devingress-ingress-nginx-controller-c2x42   1/1     Running   0          18h
pod/devingress-ingress-nginx-controller-wtmgw   1/1     Running   0          18h

NAME                                                    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE
service/devingress-ingress-nginx-controller             LoadBalancer     x.x.x.x         1.2.3.40     80:32386/TCP,443:30020/TCP   18h
service/devingress-ingress-nginx-controller-admission   ClusterIP        x.x.x.x           <none>        443/TCP                      18h

NAME                                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/devingress-ingress-nginx-controller   2         2         2       2            2           kubernetes.io/os=linux   18h

Now we have deployed two pods namely nginx with LoadBalancer service type & nginx-deploy-main with ClusterIP service type.

[~]# kubectl get all -n default
NAME                                     READY   STATUS    RESTARTS   AGE
pod/nginx-854cf6b4d7-lv5ss               1/1     Running   0          18h
pod/nginx-deploy-main-6b5457fbb5-7tg9z   1/1     Running   0          18h

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
service/nginx                  LoadBalancer   x.x.x.x        1.2.3.41       8080:31101/TCP    18h
service/nginx-deploy-main      ClusterIP      x.x.x.x          <none>           80/TCP          18h

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx               1/1     1            1           18h
deployment.apps/nginx-deploy-main   1/1     1            1           18h

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-854cf6b4d7               1         1         1       18h
replicaset.apps/nginx-deploy-main-6b5457fbb5   1         1         1       18h

Below is the ingress resource setup to access nginx-deploy-main.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: nginx-main.int.org.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-deploy-main
          servicePort: 80

And the ingress resource seems to be created correctly pointing to nginx-deploy-main service.

[~]# kubectl get ing -n default
NAME                 CLASS   HOSTS                           ADDRESS   PORTS   AGE
ingress-resource     nginx   nginx-main.int.org.com                    80      19h

[~]# kubectl describe ing/ingress-resource -n default
Name:             ingress-resource
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                           Path  Backends
  ----                           ----  --------
  nginx-main.int.org.com
                                 /   nginx-deploy-main:80 (x.x.x.x:80)
Annotations:                     kubernetes.io/ingress.class: nginx
Events:                          <none>

Outside of K8s cluster, we have nginx set up serving as reverse proxy with domain int.org.com resolution.

Below is the nginx configuration which should help me hit url http://nginx-main.int.org.com and get response but the response returned is 404.

upstream nginx-main.int.org.com {
  server 1.2.3.40:80;     ## Ingress Controller Service IP
}

server {
  listen 80;
  server_name nginx-main.int.org.com;
  location / {
    proxy_pass http://nginx-main.int.org.com;
  }
}

Now when I try to access nginx pod (not nginx-main) using its LoadBalancer Service IP with below configuration , its able to provide response and works just fine

upstream nginx.int.org.com {
  server 1.2.3.41:8080;
}

server {
  listen 80;
  server_name nginx.int.org.com;
  location / {
    proxy_pass http://nginx.int.org.com;
  }
}

Am I missing something here with regards to Ingress Controller or Resource. Port Forwarding works fine and am able to access services using the same.

This really is a blocker and any help or documentation reference would be really useful .