But, I thought that K8s provided some kind of external cluster ip that balanced the requests to the nodes from the outside. What is that IP??

• Cluster IP is internal to Cluster. Not exposed to outside, it is for intercommunication across the cluster.

• Indeed, you have LoadBanacer type of service that can do such a trick that you need, only it is dependent on cloud providers or minikube/docker edge to work properly.

I can access each pod in a node directly using their node IP

• Actually you don't access them individually that way. NodePort does a bit different trick, since it is essentially loadbalancing requests from outside on ANY exposed node IP. In a nutshell, if you hit any of node's IPs with exposed NodePort, kube-proxy will make sure that required service gets it and then service is doing round-robin through active pods, so although you hit specific node IP, you don't necessarily get pod running on that specific node. More details on that you can find here: https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0, as author there said, not technically most accurate representation, but attempt to show on logical level what is happening with NodePort exposure:

• As a sidenote, in order to do this on bare metal and do ssl or such, you need to provision ingress of your own. Say, place one nginx on specific node and then reference all appropriate services you want exposed (mind fqdn for service) as upstream(s) that can run on multiple nodes with as many nginx of their own as desired - you don't need to handle exact details of that since k8s runs the show. That way you have one node point (ingress nginx) with known IP address that is handling incoming traffic and redirecting it to services inside k8s that can run across any node(s). I suck with ascii art but will give it a try:

  (outside) -> ingress (nginx) +--> my-service FQDN (running accross nodes):
               [node-0]        |      [node-1]: my-service-pod-01 with nginx-01
                               |      [node 2]: my-service-pod-02 with nginx-02
                               |      ...
                               +--> my-second-service FQDN
                               |      [node-1]: my-second-service-pod with apache?
                               ...
  

  In above sketch you have nginx ingress on node-0 (known IP) that takes external traffic and then handles my-service (running on two pods on two nodes) and my-second-service (single pod) as upstreams. You only need to expose FQDN on services for this to work without worrying about details of IPs of specific nodes. More info you can find in documentation: https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/

  Also way better than my ansi-art is this representation from same link as in previous point that illustrate idea behind ingress:

Updated for comments

Why isn't the service load balancing the used pods from the service?

• This can happen for several reasons. Depending on how your Liveness and Readiness Probes are configured, maybe service still don't see pod as out of service. Due to this async nature in distributed system such as k8s we experience temporary lost of requests when pods get removed during, for example rolling updates and similar. Secondly, depending on how your kube-proxy was configured, there are options where you can limit it. By official documentation (https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) using --nodeport-addresses you can change node-proxy behavior. Turns out that round-robin was old kube-proxy behavior, apparently new one should be random. Finally, to exclude connection and session issues from browser, did you try this from anonymous session as well? Do you have dns cached locally maybe?

Something else, I killed the pod from node1, and when calling to node1 it didn't use the pod from node 2.

• This is a bit strange. Might be related to above mentioned probes thoug. According to official documentation this should not be the case. We had NodePort behaving inline with official documentation mentined above: and each Node will proxy that port (the same port number on every Node) into your Service. But if that is your case then probably LB or Ingress, maybe even ClusterIP with external address (see below) can do the trick for you.

if the service is internal (ClusterIP) ... does it load balance to any of the pods in the nodes

• Most definitely yes. One more thing, you can use this behavior to also expose 'load balanced' behavior in 'standard' port range as opposed to 30k+ from NodePort. Here is excerpt of service manifest we use for ingress controller.

  apiVersion: v1
  kind: Service
  metadata:
      namespace: ns-my-namespace
      name: svc-nginx-ingress-example
      labels:
          name: nginx-ingress-example
          role: frontend-example
          application: nginx-example
  spec:
      selector:
          name: nginx-ingress-example
          role: frontend-example
          application: nginx-example
      ports:
      - protocol: TCP
        name: http-port
        port: 80
        targetPort: 80
      - protocol: TCP
        name: ssl-port
        port: 443
        targetPort: 443
      externalIPs:
      - 123.123.123.123

  Note that in the above example imaginary 123.123.123.123 that is exposed with externalIPs represents ip address of one of our worker nodes. Pods running in svc-nginx-ingress-example service doesn't need to be on this node at all but they still get the traffic routed to them (and loadbalanced across the pods as well) when that ip is hit on specified port.