K
Q

Couldn't proceed with upgrade process as new nodes are not joining node group standard-workers

12/4/2020

I am trying to upgrade my kubernetes version from 1.14 to 1.15, the cluster upgrade went well but when i am trying to update the node i am seeing the message Couldn't proceed with upgrade process as new nodes are not joining node group standard-workers . I had created the nodes using eksctl.

I see the following error when i check the new node details under workloads.

runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

I also checked the tags associated with the new nodes getting spun up, they are all the same with one difference, the existing nodes have aws:ec2launchtemplate:version set to 1 and new nodes have it set to 4

I have checked the CNI plugin version and it corresponds to the latest once recommened

kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.7.5-eksbuild.1
amazon-k8s-cni:v1.7.5-eksbuild.1

Any help on how to get around this would be really helpfull.

TIA

-- opensource-developer
amazon-eks
amazon-web-services
kubernetes

Similar Questions

Ingress rule does not work with Service of type LoadBalancer
Kubernetes replace deployment fails
azure-kubernetes-service fails at creation on webconsole
Nginx request for two or more nodes in Kubernetes
Kubectl connectivity issue

2 Answers

12/4/2020

You might have to update you CNI version. Please follow the CNI upgrade tutorial from here: https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

-- Vikrant
Source: StackOverflow
12/7/2020

for others visiting the post. I found out that the issue was the url i was using to configure the kube proxy.

As per the documentation https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html, it says 

Update kube-proxy to the recommended version by taking the output from the 
previous step and replacing the version tag with your cluster's recommended
 kube-proxy version:


kubectl set image daemonset.apps/kube-proxy \
    -n kube-system \
    kube-proxy=<602401143452.dkr.ecr.us-west-2.amazonaws.com>/eks/kube-proxy:v<1.18.9>-eksbuild.1

Your account ID and Region may differ from the example above.

I misunderstood the last part of account id and substituted it with my account id, which resulted in the image not getting found.

After using the correct link with account id 602401143452, i was able to fix the issue and node group upgrade was successful.

Hope this helps, Thanks.

-- opensource-developer
Source: StackOverflow