Kubernetes tx_dropped counter increasing

12/18/2018

I'm running Kubernetes on AWS (EKS) and I'm seeing high outgoing packet loss, but no incoming packet loss. For example, notice how high the TX-DRP value for eth0 is here.

$ netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
docker0   1500        0      0      0 0             0      0      0      0 BMU
eni2b3ec  1500 20399771      0      0 0      17240493      0      0      0 BMRU
eni50c11  1500  9384173      0      0 0       9606162      0      0      0 BMRU
eni61c6c  1500 92229565      0      0 0      95948963      0      0      0 BMRU
eni693a9  1500  1684575      0      0 0       1688745      0      0      0 BMRU
eni79557  1500  6959956      0   1458 0       6977604      0    443      0 BMRU
eni9af83  1500  1049576      0      0 0       1039711      0      0      0 BMRU
enib05e7  1500 105417445      0      0 0      66847386      0      0      0 BMRU
eth0      1500 862277751      0      0 0      612433399      0 1102575      0 BMRU
eth1      1500  4342993      0      0 0       4505857      0      0      0 BMRU
eth2      1500 114179707      0      0 0      79244800      0      0      0 BMRU
lo       65536     4760      0      0 0          4760      0      0      0 LRU

The counter normally goes up very slowly so it normally isn't a problem, but I recently started performing load tests and started seeing the counter increase quickly even when doing very small load tests e.g. 1 concurrent user with apache-bench.

tcpdump shows retransmissions and duplicate acks so that confirms the dropped packets.

Cpu and memory all seem fine to me.

My questions are

  1. Why might only ongoing packets be dropped?
  2. Is there some way I can dig deeper to find the reason for the drops?
  3. Is there anything I can do (sysctl tuning?) to fix this?
-- Jesse Shieh
kubernetes
networking
performance-testing

0 Answers