We are using Kubernetes v1.9.5 on bare metal, deployed with kubespray, network driver - flannel.
When doing HTTP request from pod to another service, if that service has no endpoint, request hangs for exactly 130 seconds (I checked in via NodeJS net library and via curl).
Where this value comes from?
We noticed in while writing retries that try to establish connection to service.
Any help appreciated.
It looks like the iptables will DROP packet if its destined to a non-existent end-point and DROP will never send a message back to the requester (DROP/REJECT) . This will result in retires from requester. Retries depends on the parameter net.ipv4.tcp_syn_retries and by default its 6 in CentOS-7
When I set net.ipv4.tcp_syn_retries=1 in the minion where the requester pod runs , the timeout happens in 3 seconds instead of the 2m 7.23s
I hope this clarifies why curl or any port connect request hangs for long time if the request is for a non-existent endpoint.