I'm running Kubernetes on AWS (EKS) and I'm seeing high outgoing packet loss, but no incoming packet loss. For example, notice how high the TX-DRP
value for eth0
is here.
$ netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
docker0 1500 0 0 0 0 0 0 0 0 BMU
eni2b3ec 1500 20399771 0 0 0 17240493 0 0 0 BMRU
eni50c11 1500 9384173 0 0 0 9606162 0 0 0 BMRU
eni61c6c 1500 92229565 0 0 0 95948963 0 0 0 BMRU
eni693a9 1500 1684575 0 0 0 1688745 0 0 0 BMRU
eni79557 1500 6959956 0 1458 0 6977604 0 443 0 BMRU
eni9af83 1500 1049576 0 0 0 1039711 0 0 0 BMRU
enib05e7 1500 105417445 0 0 0 66847386 0 0 0 BMRU
eth0 1500 862277751 0 0 0 612433399 0 1102575 0 BMRU
eth1 1500 4342993 0 0 0 4505857 0 0 0 BMRU
eth2 1500 114179707 0 0 0 79244800 0 0 0 BMRU
lo 65536 4760 0 0 0 4760 0 0 0 LRU
The counter normally goes up very slowly so it normally isn't a problem, but I recently started performing load tests and started seeing the counter increase quickly even when doing very small load tests e.g. 1 concurrent user with apache-bench.
tcpdump
shows retransmissions and duplicate acks so that confirms the dropped packets.
Cpu and memory all seem fine to me.
My questions are