Kubernetes controller manager pod not updating

11/26/2018

I am trying to add some extra flags to my kubernetes controller manager and I am updating the flags in the /etc/kubernetes/manifests/kube-controller-manager.yaml file. But the changes that I am adding are not taking effect. The kubelet is detecting changes to the file and is restarting the pods but once restarted they come back with the old flags.

Any ideas?

-- devops84uk
kube-controller-manager
kubernetes

2 Answers

11/26/2018

The reason behind that is the container is started with the old flags, when you pass new flags and kube-controller-manager pod is restarted(pod restart doesn't mean container restart) but the kube-controller-manager container is still using old flags.

Check it using following command:

docker ps --no-trunc | grep "kube-controller-manager --"
dcc828aa22aae3c6bb3c4ba31d0cfcac669b9c47e4cf50af580ebbb334bfea9f   sha256:40c8d10b2d11cbc3db2e373a5ffce60dd22dbbf6236567f28ac6abb7efbfc8a9                                          "kube-controller-manager --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --pod-eviction-timeout=30s --leader-elect=true --use-service-account-credentials=true --controllers=*,bootstrapsigner,tokencleaner --root-ca-file=/etc/kubernetes/pki/ca.crt --address=127.0.0.1 --kubeconfig=/etc/kubernetes/controller-manager.conf --service-account-private-key-file=/etc/kubernetes/pki/sa.key --allocate-node-cidrs=true --cluster-cidr=192.168.13.0/24 --node-cidr-mask-size=24"                                                  

Once you update the flags in /etc/kubernetes/manifests/kube-controller-manager.yaml file, restart the docker container of kube-controller-manager and changes will take effect. You can use following command to restart the kube-controller-manager container:

docker restart $(docker ps --no-trunc | grep "kube-controller-manager --" | awk '{print $1}')

Hope this helps.

-- Prafull Ladha
Source: StackOverflow

11/26/2018

So it seems that any file under /etc/kubernetes/manifests is loaded by the kubelet. So when I was adding the new flags I was taking a backup of the existing file with a .bak extension but kubelet was still loading the .bak file instead of the new .yaml file. Seems to me thats a bug. Anyways, happy to have spotted the error.

-- devops84uk
Source: StackOverflow