Load distribution: All HTTP requests are getting redirected to a single pod in a k8 cluster

11/20/2020

I have created a very simple spring boot application with only one REST service. This app is converted into a docker image ("springdockerimage:1") and deployed in the Kubernetes cluster with 3 replicas. Contents of my "Deployment" definition is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: springapp
  labels:
   app: distributiondemo
spec:
 selector:
   matchLabels:
     app: distributiondemo
 replicas: 3
 template:
   metadata:
     labels:
       app: distributiondemo
   spec:
     containers:
      - name: spring-container
        image: springdockerimage:1

I have created service for my above deployment as follows:

apiVersion: v1
kind: Service
metadata:
  name: springservice
  labels:
    app: distributiondemo
spec:
  selector:
    app: distributiondemo
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
    name: spring-port
    nodePort: 32000
  type: NodePort

After deploying both the above YAML(deployment and service) files, I noticed that everything has been deployed as expected i.e., 3 replicas are created and my service is having 3 endpoints as well. Below screenshot is the proof of the same: enter image description here

Since I am using minikube for my local testing, I am port forwarding and accessing the application as kubectl port-forward deployment.apps/springapp 40002:8080 .

But one thing I noticed is that all my HTTP requests are getting redirected to only one pod.

while true ; do curl http://localhost:40002/docker-java-app/test ;done

I am not getting where exactly I am doing it wrong. Any help would be appreciated. Thank you.

-- Hemanth H L
kube-proxy
kubernetes
kubernetes-pod
load-balancing
minikube

1 Answer

11/20/2020

The loadbalancing might not work with port-forwarded ports as it might be directly redirecting traffic to pod (read more here). The K8s service is the feature will give you that loadbalancing capability.

So you can try either of below instead

  1. Use http://your_service_dns_name:8080/docker-java-app/test
  2. Use http://service_cluster_ip:8080/docker-java-app/test
  3. Use http://any_host_ip_from_k8s_cluster:32000/docker-java-app/test

Option 1 and 2 works only if you are accessing those urls from a host which is part of K8s cluster. Option 3 just needs connectivity to target host and port, from the host you are accessing url.

-- Syam Sankar
Source: StackOverflow