I have setup a kubernetes cluster with kubeamd; One control-plane and a worker node.
Everything worked fine. Then I setup a Squid proxy on the worker node and in the kubelet config I have set http_proxy=http://127.0.0.1:3128 essentially asking kubelet to use the proxy to communicate to the control-plane.
I see, using tcpdump, network packets landing on the control plane from worker node, and I am able to issue the following command from worker as well;
kubectl get no --server=https://10.128.0.63:6443
NAME STATUS ROLES AGE VERSION
k8-cp Ready master 6d6h v1.17.0
k8-worker NotReady <none> 6d6h v1.17.2but the worker status always remains NotReady. What might I be doing wrong?
I am using Flannel here for networking.
P.S. I have exported http_proxy=http://127.0.0.1:3128 as an env variable as well before issuing
kubectl get no --server=https://10.128.0.63:6443from the worker node.
If it matters here is the node status;
kubectl describe no k8-worker
Name: k8-worker
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=k8-worker
kubernetes.io/os=linux
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"fe:04:d6:53:ef:cc"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 10.128.0.71
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 29 Jan 2020 08:08:33 +0000
Taints: node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: k8-worker
AcquireTime: <unset>
RenewTime: Thu, 30 Jan 2020 11:51:24 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Thu, 30 Jan 2020 11:48:25 +0000 Thu, 30 Jan 2020 11:52:08 +0000 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Thu, 30 Jan 2020 11:48:25 +0000 Thu, 30 Jan 2020 11:52:08 +0000 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Thu, 30 Jan 2020 11:48:25 +0000 Thu, 30 Jan 2020 11:52:08 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Thu, 30 Jan 2020 11:48:25 +0000 Thu, 30 Jan 2020 11:52:08 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 10.128.0.71
Hostname: k8-worker
Capacity:
cpu: 2
ephemeral-storage: 104844988Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7493036Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 96625140781
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7390636Ki
pods: 110
System Info:
Machine ID: 3221f625fa75d20f08bceb4cacf74e20
System UUID: 6DD87A9F-7F72-5326-5B84-1B3CBC4D9DBE
Boot ID: 7412bb51-869f-40de-8b37-dcbad6bf84b4
Kernel Version: 3.10.0-1062.9.1.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://1.13.1
Kubelet Version: v1.17.2
Kube-Proxy Version: v1.17.2
PodCIDR: 10.244.1.0/24
PodCIDRs: 10.244.1.0/24
Non-terminated Pods: (3 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default nginx-86c57db685-fvh28 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6d20h
kube-system kube-flannel-ds-amd64-b8vbr 100m (5%) 100m (5%) 50Mi (0%) 50Mi (0%) 6d23h
kube-system kube-proxy-rsr7l 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6d23h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 100m (5%) 100m (5%)
memory 50Mi (0%) 50Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>Link to kubelet logs on worker:
The Kube-controller-manager/node-controller is responsible for monitoring the health of the nodes monitoring the endpoint "/healthz" exposed by kubelet.
So far you have configured a one way communication over proxy (from Node to Master).
You need to do it for other components, especially Kube-controller-manager. This way you enable two way communication over HTTP Proxy.
This is achievable by specifying HTTP_PROXY on KUBEADM INIT:
$ sudo http_proxy=192.168.1.20:3128 kubeadm init
Learn more here: Kubadm Issue 182
You will see some output like this:
kubeadm@lab-1:~$ sudo http_proxy=192.168.1.20:3128 kubeadm init
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
[WARNING HTTPProxy]: Connection to "https://10.156.0.6" uses proxy "http://192.168.1.20:3128". If that is not intended, adjust your proxy settings
[WARNING HTTPProxyCIDR]: connection to "10.96.0.0/12" uses proxy "http://192.168.1.20:3128". This may lead to malfunctional cluster setup. Make sure that Pod and Services IP ranges specified correctly as exceptions in proxy configurationLearn more here: Kubeadm Issue 324.