How can I get kubectl to recognize the newly scaled az aks nodepool nodes?

8/12/2020

I updated my Azure AKS nodepool size from within the Azure Portal to go from 2 to 4 nodes. When I run az aks nodepool show ..., I see that the count has correctly been updated. However, when I run kubectl get nodes, I still only see the two nodes that previously existed.

According to the Kubernetes documentation on node management,

There are two main ways to have Nodes added to the API server : 1. The kubelet on a node self-registers to the control plane 2. You, or another human user, manually add a Node object

(Emphasis mine)

My expectation, therefore, is that having scaled up my node pool, these new nodes should automatically register, and kubectl get nodes should just pick them up, but this appears to not be the case.

Now that my nodepool has more nodes, how do I get my AKS cluster to recognize and utilize them? Once kubectl get nodes shows them, will applying an updated manifest (with more replicas) be all I need to do to use the additional hardware?

-- user655321
azure-aks
kubernetes

1 Answer

8/12/2020

It's difficult to see without access to your setup. But you can see:

  • Check that the control plane hasn't been automatically upgraded to a new version that is incompatible with the kubelet version in your nodepool when it registers with the cluster. (Best if the versions match)
  • Connect to the nodes that are not registering (ssh) and check the logs as to why the kubelet is not starting. i.e systectl status kubelet.
  • Check that you can connect to the port (i.e 8443) and IP address where your kube-apiserver is listening on from these nodes that are not registering. i.e curl <ip-address>:8443

Possible solution:

  • Upgrade the VM image of your node pool to use one compatible with the control plane.
  • Remove firewall rule preventing your nodes accessing the kube-apiserver

will applying an updated manifest (with more replicas) be all I need to do to use the additional hardware?

Should work.

✌️

-- Rico
Source: StackOverflow