The etcdserver throws warnings of request 'took to long'

12/6/2021

I have an MKE cluster with three manager node. The etcd is giving the following warnings

2021-12-06 10:03:10.624622 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:8" took too long (157.50303ms) to execute
2021-12-06 10:03:24.987502 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:442" took too long (127.110289ms) to execute
2021-12-06 10:03:26.302979 W | etcdserver: request "header:<ID:16726225580315543268 username:\"system:ucp:v2d7rih1ql0sztk06dnu37a58\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:352278653 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:218 >> failure:<>>" with result "size:22" took too long (101.390612ms) to execute
2021-12-06 10:03:26.498919 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:8" took too long (141.572923ms) to execute
2021-12-06 10:03:37.589988 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:296" took too long (141.997165ms) to execute
2021-12-06 10:04:03.211705 W | etcdserver: read-only range request "key:\"/registry/networkpolicies/\" range_end:\"/registry/networkpolicies0\" count_only:true " with result "range_response_count:0 size:8" took too long (108.336554ms) to execute
2021-12-06 10:04:08.707179 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:8" took too long (102.900861ms) to execute
2021-12-06 10:04:09.145791 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:296" took too long (253.282239ms) to execute
2021-12-06 10:04:09.834741 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:460" took too long (149.503672ms) to execute
2021-12-06 10:04:25.053962 W | etcdserver: read-only range request "key:\"/registry/persistentvolumeclaims/tme-ns-cms-prd/tme-pvc-tmeestore\" " with result "range_response_count:1 size:1201" took too long (209.981256ms) to execute
2021-12-06 10:05:30.778453 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:460" took too long (253.111751ms) to execute

We already increased the CPUs and memory on these three manager nodes, without any impact.

I suppose that disk IO is very important and can result in these warnings if we have to high latency?

Can network between the three etcd instances cause this too? Or are these requests pure locally ?

-- WalterH
etcd
kubernetes

0 Answers