I am new to the Kubernetes cluster and trying to setup my very first kubernetes cluster and very first load balanced service. I have a hybrid environment which contains two Windows and two CentOS nodes + 1 CentOs Master node

As far as Kubernetes goes, everything is working fine. I deployed nginx-ingress controller as daemon set which runs only on Linux servers. I followed the following tutorial: https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md with only one difference that in deployments/deployment/nginx-ingress.yml, I restricted to run it on Linux machines using nodeselector

I created a deployment with 2 replicas and service for the deployment. I also created ingress resource But when I create a service, it always goes to Pod on one node. Here is my code:

Service and Deployment:

apiVersion: v1
kind: Service
metadata:
  name: myservice-1
  labels:
    app: myservice-1
spec:
  ports:
    # the port that this service should serve on
  - port: 9034
    targetPort: 80
  selector:
    app: myservice-1 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myservice-1
  name: myservice-1
spec:
  selector:
    matchLabels:
      app: myservice-1
  replicas: 2
  template:
    metadata:
      labels:
        app: myservice-1
        run: with-creds        
      name: myservice-1
    spec:
      containers:
      - name: myservice-1
        securityContext:
          windowsOptions:
            gmsaCredentialSpecName: gmsa-dev-credspec
        image: secure-myprivaterepo:443/docker/library/myservice-1        
      nodeSelector:
        beta.kubernetes.io/os: windows

And here is my ingress resource

Ingress Resource:

kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: myservice-1-ingress  
spec:
  rules:
    - host: myservice1
      http:
        paths:
          - backend:
              serviceName: myservice-1
              servicePort: 9034

I see that both my replicas are working no two different Windows node. I use HAProxy to roundrobin the request to my two Linux nodes so that it can hit the nginx-ingress controller running on those nodes. From logs, I can see that request is being routed to both of these servers in (kind of) round robin fashion. Here is my HAProxy config.

HAProxy front end and backend config

HA Proxy Defaults:

More Details (requested by PjoterS):

• CNI Used: Flannel (I used documentation on Microsoft site to configure Flannel: https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/network-topologies) and used host-gw configuration
• I have used the following document from Microsoft to join my windows node to my Kubernetes cluster: https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/joining-windows-workers?tabs=ManagementIP#joining-a-flannel-cluster
• I have used kubeadm on bare metal CentOS machines to create a cluster
• I can't ping From one pod to another pod living on different nodes
• Pods are not scheduled on Linux node so I can't curl to my Windows node. These are Windows Full Framework (mcr.microsoft.com/dotnet/framework/aspnet) images and can't be scheduled on Linux servers
• Out of curiosity, I scaled my deployment up to 4 replicas so each of my two windows servers can get two pods. Now when I ping one pod on to another on the same node, ping works just fine. So It seems that pods on different nodes can't be reached by each other.

Problem:

I made a host entry currently in my local machine pointing to my HAProxy iP and visit site http://myservice1 and my windows service is working fine but when I exec to each pod and look at the application log, I see that log is generated on just one pod and pod on the other node hasn't even being called once. I tried more than 100 requests and every time, node 1 responded and node 2 is not invoked at all.

Do I have to do anything specific so that ingress controller really forwards traffic in round robin fashion? I tried searching on blogs but hasn't find any answers at all