K
Q

how to test performance of kubernetes cluster network?

10/2/2019

I've a kubernetes cluster in aws, created with kops. I'm trying to evaluate network performance of the kubernetes cluster. I'm using default kubernetes CNI kubenet.

networking:
    kubenet: {}

I've used kubernetes-perf-test's net-perf to evaluate kubernetes cluster's network performance.

I've used below command to run the test.

go run ./launch.go --iterations 2

generated output

MSS                                          , Maximum, 96, 160, 224, 288, 352, 416, 480, 544, 608, 672, 736, 800, 864, 928, 992, 1056, 1120, 1184, 1248, 1312, 1376, 1440,
1 iperf TCP. Same VM using Pod IP            ,36736.000000,2101,25486,36736,35636,36115,35631,6801,6544,7146,6565,6320,7252,6363,5707,5855,6034,36058,34098,36713,35625,34571,6341,
2 iperf TCP. Same VM using Virtual IP        ,30203.000000,5531,6072,5963,5294,6047,5879,5412,5456,5844,28499,29306,29289,30203,28516,5987,5460,5833,6029,4920,5027,5111,4700,
3 iperf TCP. Remote VM using Pod IP          ,664.000000,0,0,102,134,156,196,229,255,266,189,214,240,235,267,314,294,403,531,612,617,664,657,
4 iperf TCP. Remote VM using Virtual IP      ,778.000000,0,0,134,163,202,240,287,313,338,414,481,477,512,610,605,559,604,578,572,679,778,697,
5 iperf TCP. Hairpin Pod to own Virtual IP   ,29522.000000,974,547,19748,22219,24301,24043,25328,24531,4508,4471,5699,4339,4199,5001,5504,4974,15043,28658,29522,28902,28574,28956,
6 iperf UDP. Same VM using Pod IP            ,1592.000000,1592,
7 iperf UDP. Same VM using Virtual IP        ,1008.000000,1008,
8 iperf UDP. Remote VM using Pod IP          ,0.000000,0,
9 iperf UDP. Remote VM using Virtual IP      ,0.000000,0,
10 netperf. Same VM using Pod IP             ,3516.740000,3516.74,
11 netperf. Same VM using Virtual IP         ,0.000000,0.00,
12 netperf. Remote VM using Pod IP           ,422.430000,422.43,
13 netperf. Remote VM using Virtual IP       ,0.000000,0.00,

I'm not sure what actually this output means. How can I determine with this value, whether my k8s cluster's network is performing okay or not?

Also, I've tried to plot this data.enter image description here enter image description here

-- Abu Hanifa
amazon-web-services
kops
kubernetes

Similar Questions

Can i insert secrets into deployment.yml using Azure KeyVault?
kubernetes: can't deploy jenkins images with persistent volume with RW access
Decrease Prometheus scraping interval on k8s for one Pod
CreateContainerConfigError while migrating Kubernetes cluster
Rancher - cluster node has disk pressure

1 Answer

10/3/2019

This script is testing iperf/netperf transfer speed in MSS testpoint in 5 scenarios for TCP, UDP and NetPerf.

Honestly all you have described in provided Github links.

Iperf is a commonly used network testing tool that can create TCP/UDP data streams and measure the throughput of a network that is carrying them. It allows the user to set various parameters that can be used for testing a network, or alternatively for optimizing or tuning a network.

Netperf is also another good network testing tool, which is also used by the PerfKitBenchmark suite to test performance and benchmark the various cloud providers to each other.

More information you can find here

On the Github you have 2x5 ipetf scenarios for TCP and UPD and 5 scenarios for netperf.

The 5 major network traffic paths are combination of Pod IP vs Virtual IP and whether the pods are co-located on the same node/VM versus a remotely located pod.
- Same VM using Pod IP

Same VM Pod to Pod traffic tests from Worker 1 to Worker 2 using its Pod IP.
- Same VM using Cluster/Virtual IP

Same VM Pod to Pod traffic tests from Worker 1 to Worker 2 using its Service IP (also known as its Cluster IP or Virtual IP).
- Remote VM using Pod IP

Worker 3 to Worker 2 traffic tests using Worker 2 Pod IP.
- Remote VM using Cluster/Virtual IP

Worker 3 to Worker 2 traffic tests using Worker 2 Cluster/Virtual IP.
- Same VM Pod Hairpin to itself using Cluster IP

Regarding CSV output

MMS - Maximum Segment Size. More information about it can be found here.

Example TCP MMS The TCP protocol includes a mechanism for both ends of a connection to negotiate the maximum segment size (MSS) to be used over the connection. Each end uses the OPTIONS field in the TCP header to advertise a proposed MSS. The MSS that is chosen is the smaller of the values provided by the two ends.

The purpose of this negotiation is to avoid the delays and throughput reductions caused by fragmentation of the packets when they pass through routers or gateways and reassembly at the destination host.

The value of MSS advertised by the TCP software during connection setup depends on whether the other end is a local system on the same physical network (that is, the systems have the same network number) or whether it is on a different (remote) network.

So for each scenario (Maximum Segment Size), for protocol, script is measuring transfer speed per MSS datapoint in Mbit/sec. This output will have better visibility if you paste it to excel and use "Text to Columns" divided by comma.

In graph you have the same values, just for better visibility.

-- PjoterS
Source: StackOverflow