Have an EKS Cluster that has an ELB along with 3 worker nodes attached to it. The application is running within the container on 30590. Have configured health check on the same port 30590. Kube-proxy is listening to this port. But the worker nodes are OutOfService behind the ELB.
pkts bytes target prot opt in out source destination
13884 826K KUBE-EXTERNAL-SERVICES all -- any any anywhere anywhere ctstate NEW /* kubernetes externally-visible service portals */
2545K 1268M KUBE-FIREWALL all -- any any anywhere anywhere
Chain FORWARD (policy ACCEPT 92 packets, 28670 bytes)
pkts bytes target prot opt in out source destination
1307K 409M KUBE-FORWARD all -- any any anywhere anywhere /* kubernetes forwarding rules */
1301K 409M DOCKER-USER all -- any any anywhere anywhere
Chain OUTPUT (policy ACCEPT 139 packets, 12822 bytes)
pkts bytes target prot opt in out source destination
349K 21M KUBE-SERVICES all -- any any anywhere anywhere ctstate NEW /* kubernetes service portals */
2443K 222M KUBE-FIREWALL all -- any any anywhere anywhere
Chain DOCKER (0 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (0 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- any any anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (0 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- any any anywhere anywhere
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
1301K 409M RETURN all -- any any anywhere anywhere
Chain KUBE-EXTERNAL-SERVICES (1 references)
pkts bytes target prot opt in out source destination
Chain KUBE-FIREWALL (2 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- any any anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
Chain KUBE-FORWARD (1 references)
pkts bytes target prot opt in out source destination
3 180 ACCEPT all -- any any anywhere anywhere /* kubernetes forwarding rules */ mark match 0x4000/0x4000
Chain KUBE-SERVICES (1 references)
pkts bytes target prot opt in out source destination
12:41:44.217236 IP ip-192-168-186-107.ec2.internal.22580 > ip-x-x-x-.ec2.internal.30590: Flags [S], seq 3790958206, win 29200, options [mss 1460,sackOK,TS val 10236779 ecr 0,nop,wscale 8], length 0
12:41:44.217834 IP ip-x-x-x-.ec2.internal.30590 > ip-192-168-186-107.ec2.internal.22580: Flags [R.], seq 0, ack 3790958207, win 0, length 0
Looks like the EKS nodes are sending TCP RSTs to the ELB as this is why they are failing the ELB healthchecks. Can anyone help me in troubleshooting the issue?
Found the solution :) The issue was with the replicationcontroller.json file, that I had mentioned a wrong port to be exposed, & trying to connect on a different port.