I am trying to understand the Nginx-Ingress on the K8S cluster.

I set up the nginx-ingress controller based on the instructions here

My cluster has 3 nodes kubernetes-master, kubernetes-node1, kubernetes-node2

I have an IoTPoD running with 1 replica (kubernetes-node1). I have created a Cluster IP service for accessing this pod via rest. Below is the manifest.

apiVersion: v1
kind: Namespace
metadata:
  name: myiotgarden
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ioteventshandler
  namespace: myiotgarden
  labels:
    app:  ioteventshandler
spec:
  replicas: 2
  selector:
    matchLabels:
      app:  ioteventshandler
  template:
    metadata:
      labels:
        app:  ioteventshandler
    spec:
      containers:
      - name:  ioteventshandler
        image: 192.168.56.105:5000/ioteventshandler:latest
        resources:
          limits:
            memory: "1024M"
          requests:
            memory: "128M"
        imagePullPolicy: "Always"
---
apiVersion: v1
kind: Service
metadata:
  name: iotevents-service
  namespace: myiotgarden
  labels:
    app: iotevents-service
spec:
  selector:
    app: ioteventshandler
  ports:  
  - port: 8080
    protocol: TCP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ioteventshandler-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
  namespace: myiotgarden
spec:
  rules:
  - host: kubernetes-master
    http:
      paths:
      - path: /iotEvents/sonoff
        backend:
          serviceName: iotevents-service
          servicePort: 8080
  - host: kubernetes-node1
    http:
      paths:
      - path: /iotEvents/sonoff
        backend:
          serviceName: iotevents-service
          servicePort: 8080
  - host: kubernetes-node2
    http:
      paths:
      - path: /iotEvents/sonoff
        backend:
          serviceName: iotevents-service
          servicePort: 8080

I have a haproxy running on Kmaster node. which has been configured with both the knode1 and knode2 as worker nodes.

frontend http_front
  bind *:80
  stats uri /haproxy?stats
  default_backend http_back

backend http_back
  balance roundrobin
  server worker 192.168.56.207:80
  server worker 192.168.56.208:80

When there are 2 replicas of the ioteventshandler running, the below command works fine.

curl -kL http://kubernetes-master/iotEvents/sonoff/sonoff11

I get back the response perfectly.

However, when I reduce the replicas to 1. the curl command intermittently returns 502 Bad Gateway. I am assuming this happens when the haproxy forwards the request to knode2 where a replica of ioteventshandler is not running.

Question: Is there a way that the ingress controller on knode2 forwards it to knode1? And how do we do it?

Thank you.