All of the nodes in our AWS kubernetes cluster (Server Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.6", GitCommit:"388061f00f0d9e4d641f9ed4971c775e1654579d", GitTreeState:"clean"}) are getting the following messages sent to /var/log/syslog which are filling the disk very quickly (32GB in about 24 hours).
Dec 4 03:13:36 ubuntu kube-proxy[15171]: I1204 03:13:36.961584 15171 proxysocket.go:130] Accepted TCP connection from 172.30.0.164:58063 to 172.30.0.39:33570
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.961775 15171 proxysocket.go:99] Dial failed: dial tcp 10.244.0.7:5000: connection refused
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.961888 15171 proxysocket.go:99] Dial failed: dial tcp 10.244.2.9:5000: connection refused
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.962104 15171 proxysocket.go:99] Dial failed: dial tcp 10.244.0.7:5000: connection refused
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.962275 15171 proxysocket.go:99] Dial failed: dial tcp 10.244.2.9:5000: connection refused
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.962299 15171 proxysocket.go:133] Failed to connect to balancer: failed to connect to an endpoint.
Dec 4 03:13:36 ubuntu kube-proxy[15171]: I1204 03:13:36.962380 15171 proxysocket.go:130] Accepted TCP connection from 172.30.0.87:29540 to 172.30.0.39:33570
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.962630 15171 proxysocket.go:99] Dial failed: dial tcp 10.244.0.7:5000: connection refused
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.962746 15171 proxysocket.go:99] Dial failed: dial tcp 10.244.2.9:5000: connection refused
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.962958 15171 proxysocket.go:99] Dial failed: dial tcp 10.244.0.7:5000: connection refused
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.963084 15171 proxysocket.go:99] Dial failed: dial tcp 10.244.2.9:5000: connection refused
Dec 4 03:13:36 ubuntu kube-proxy[15171]: E1204 03:13:36.963105 15171 proxysocket.go:133] Failed to connect to balancer: failed to connect to an endpoint.We created the cluster using export KUBERNETES_PROVIDER=aws; curl -sS https://get.k8s.io | bash if that is relevant.
Can anyone point me into the right direction as to the cause?
port 5000 is used by the local docker registry usually. It is an add-on though. Is your cluster pulling images from that local registry? If so, is it working? how is it setup?
this link may help figure your config issues:
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry