Connect local node to kops AWS cluster

1/21/2020

I want to connect a local node (not a cloud one) to a kops-created cluster on AWS. I followed the suggested approach in Kubernetes: Combining a Kops cluster to an on-premise Kubeadm cluster. Below are my kubelet options:

DAEMON_ARGS="\
--allow-privileged=true \
--cgroup-root=/ \
--cgroup-driver=systemd \
--cluster-dns=${CLUSTER_DNS} \
--cluster-domain=cluster.local \
--enable-debugging-handlers=true \
--eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5% \.available<10%,imagefs.inodesFree<5% \
--feature-gates=DevicePlugins=true,ExperimentalCriticalPodAnnotation=true \
--hostname-override=my-node \
--kubeconfig=/var/lib/kubelet/kubeconfig \
--network-plugin-mtu=9001 \
--network-plugin=kubenet \
--node-labels=KubernetesCluster=${CLUSTER_NAME}, kubernetes.io/cluster/${CLUSTER_NAME}=owned,kubernetes.io/role=node,node-role.kubernetes.io/node= \
--node-ip=${NODE_IP} \
--non-masquerade-cidr=${NON_MASQUERADE_CIDR} \
--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \
--pod-manifest-path=/etc/kubernetes/manifests \
--register-schedulable=true \
--v=2 \
--cni-bin-dir=/opt/cni/bin/ \
--cni-conf-dir=/etc/cni/net.d/ \
--cni-bin-dir=/opt/cni/bin/"

When I start kubelet on my local node, it successfully connects to the kube-apiserver and registers the node. However, it then repeatedly fails on updating node status:

E0121 23:08:35.135858   18352 kubelet_node_status.go:383] Error updating node status, will retry: error getting node "my-node": nodes "my-node" not found
E0121 23:08:35.191611   18352 kubelet_node_status.go:383] Error updating node status, will retry: error getting node "my-node": nodes "my-node" not found
...
E0121 23:08:35.359480   18352 kubelet_node_status.go:375] Unable to update node status: update node status exceeds retry count
E0121 23:08:35.823944   18352 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "my-node" not found

Upon checking the kube-controller-manager logs (/var/log/kube-controller-manager.log on the master node), I found that kube-controller-manager is deleting my-node because it can't be found within the cloud provider, aws:

I0121 23:08:25.722214       1 node_controller.go:769] Deleting node (no longer present in cloud provider): my-node
I0121 23:08:25.722248       1 controller_utils.go:197] Recording Deleting Node my-node because it's not present according to cloud provider event message for node my-node

Is there a way to disable this cloud provider check for my-node so it doesn't get deleted by kube-controller-manager? I still want to be able to run some nodes in AWS so don't want to clear the cloud-provider flag in kube-controller-manager.

-- Wycan
amazon-web-services
kops
kubelet
kubernetes

0 Answers