When attempting to drain a node on an AKS K8s cluster using:
kubectl drain ${node_name} --ignore-daemonsetsI get the following error:
"The Node \"aks-agentpool-xxxxx-0\" is invalid: []: Forbidden: node updates may only change labels, taints, or capacity (or configSource, if the DynamicKubeletConfig feature gate is enabled)"Is there something extra that needs to be done on AKS nodes to allow draining?
(Context: This is part of an automation script I'm writing to drain a kubernetes node for maintenance operations without downtime, so the draining is definitely a prerequisite here)
An additional troubleshooting note:
This command is being run via Ansible's "shell" module, but when the command is run directly in BASH, it works fine.
Further, the ansible is being run via a Jenkins pipeline. Debug statements seem to show:
This command is being run via Ansible's "shell" module, but when the command is run directly in BASH, it works fine.
Further, the ansible is being run via a Jenkins pipeline.
It's good that you added this information because it totally changes the perspective from which we should look at the issue you experience.
For debugging purposes instead of running your command, try to run:
kubectl auth can-i drain node --all-namespacesboth directly in bash shell as well as via Ansible's shell module
It should at least give you an answer if this is not a permission issue.
Other commands that you may use to debugging in this case are:
ls -l .kube/config
cat .kube/config
whoami
Last one to make sure that Ansible uses the same user. If you already know that it uses different user, try to run the script as the same user you use for running it in a bash shell.
Once you check this, we can continue the debugging process.