After spending half a day digging the web, I'm still not able to find the reason why my worker nodes are not getting proper labels. I have my worker nodes in Auto scaling group in AWS and I'm using AWS cloud provider with Kubeadm to provision my cluster. After all my master nodes and worker nodes come up, I don't see proper labels like availability zone assigned to worker nodes which are present in the master nodes as seen below:
# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
ip-10-100-128-9.ec2.internal Ready <none> 5h12m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-128-9.ec2.internal,kubernetes.io/os=linux
ip-10-100-148-5.ec2.internal Ready <none> 5h12m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-148-5.ec2.internal,kubernetes.io/os=linux
ip-10-100-164-199.ec2.internal Ready <none> 5h12m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-164-199.ec2.internal,kubernetes.io/os=linux
ip-10-100-3-145.ec2.internal Ready master 5h15m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1c,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-3-145.ec2.internal,kubernetes.io/os=linux,node-role.kubernetes.io/master=
ip-10-100-3-55.ec2.internal Ready master 5h15m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1a,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-3-55.ec2.internal,kubernetes.io/os=linux,node-role.kubernetes.io/master=
ip-10-100-3-76.ec2.internal Ready master 5h15m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1b,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-3-76.ec2.internal,kubernetes.io/os=linux,node-role.kubernetes.io/master=I somehow doubt this to be the reason why I can't create a PVC for my PV which I want to attach to my docker registry pod.
# kubectl describe pod docker-registry-5b66dd644d-66f7k -n default | grep -A4 -i Events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 84s (x11 over 14m) default-scheduler 0/6 nodes are available: 3 node(s) had taints that the pod didn't tolerate, 3 node(s) had volume node affinity conflict.
Warning FailedScheduling 79s (x3 over 82s) default-scheduler 0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.I see my storageclass and PV created and can verify the same volume ID on AWS. I also manually tried attaching a volume to my worker nodes using AWS CLI where I was successful but not through PVC. Please advise what's getting wrong here. Thanks in Advance.
The error states node(s) had taints that the pod didn't tolerate. You must either remove those taints from the nodes using kubectl taint nodes node_name taint- or add tolerations to the objects you create.
The Issue was with the kubelet service file. By default, kubeadm doesn't add AWS cloud provider information in the kubelet unit file. Manual addition of parameter KUBELET_EXTRA_ARGS=--cloud-provider=aws inside /etc/sysconfig/kubelet file solved the issue and labels are being assigned automatically by AWS.