Connectivity issues between client to k8s cluster [TCP-reset-by-server]

6/22/2019

Last week we have migrated our whole network infrastructure to new equipment. We also replaced our Fortigate 200D to two new Palo Alto 850.

We created a new IPSec between our office into Azure Gateway following this guide: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Cm6WCAS

We manage to access all our dev environment at Azure via our new IPSec but we are facing an issue downloading files from k8s nodes.

For example: I have TeamCity hosted on k8s. When I download artifacts from my TeamCity node, after 2-3 minutes I get 'Failed - Network error' in chrome, then the download, stops immediately - and on the firewall, I see the session was RST by server 'tcp-rst-from-server'.

I troubleshoot our IPSec between our Palo Alto and Azure with Palo Alto engineers and Azure engineers and we couldn't find any miss-configuration on our Palo Alto.

I created a new Windows instance under Azure with IIS and uploaded a big file to the server so I can download simultaneity from k8s and from a normal instance.

So after 2 minutes, I get TCP-rst-from server on the k8s URL but the direct server still downloading, no drops.

Also, we don't see any drops on our side. So the only change we made in this whole flow is changing our Firewall to another vendor.

Is anyone having any guess?

Thanks all!

-- NexusIT
firewall
kubernetes
networking
nginx-ingress
vpn

1 Answer

6/24/2019

I'm afraid your test might be not too reliable (if to suspect network related issue e.g NIC), as your newly created VM and AKS cluster are not for sure in the same virtual network.

Instead of creating a new VM, re-use existing Worker node, and expose TeamCity via NodePort service. This will allow you to eliminate any possible proxy server between client and web server (IIS), if any exists in your AKS env.

It would be worth adding any details about your AKS cluster setup (Azure CNI in use ?), and the main thing how do you expose your TeamCity application right now, via NGINX ingress controller or accessing it directly on PODs individual IP address ?

-- Nepomucen
Source: StackOverflow