Accessing service is too much slower than accessing inside container

10/17/2019

I have a k8s cluster, and the version of k8s is v1.12.1. One of the services is below:

NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/flask   NodePort   10.99.101.226   <none>        8090:32010/TCP   15h

When accessing the service with CLUSTER-IP inside the cluster, it is very slow compared to accessing inside the pod container.

curl  10.99.101.226:8090/test_json > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2316k  100 2316k    0     0   263k      0  0:00:08  0:00:08 --:--:--  216k
kubectl exec -it flask-example-77dbd59654-2g857 -- sh
/app # curl localhost:8090/test_json > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2316k  100 2316k    0     0  75.3M      0 --:--:-- --:--:-- --:--:-- 75.3M

The former speed is only 216K, but the latter speed can up to 75.3M, I wonder to know what cause the difference.

Is there anyone can help me? thanks.

-- leo
kubernetes

1 Answer

10/17/2019

There's the latency of your network connection.

When you access it by exec'ing into the pod, there's no hop at all as you are on the same host and network, but the NodePort service expose your app through your node, so your connection will need to go first from your machine to the Node and then bypassed to the pod.

[your machine] <==> [NODE]:32010 <==> [POD]:8090

To have a clear view on what's happening, you can use --trace and you should be able to see the hops:

curl  --trace trace.log 10.99.101.226:32010/test_json > /dev/null

then check the file trace.log

To avoid hit the node and access the pod directly, you can use a headless service. You can do that by changing the service type to ClusterIP and set clusterIP: None on the spec

-- Pedreiro
Source: StackOverflow