K
Q

Failed to update lock: Operation cannot be fulfilled on endpoints

9/3/2019

I am installing kubernetes controller manager just from it's binary file:

cp -r /data/k8s/package/kubernetes/server/kubernetes/server/bin /usr/local/bin

When I start kubernetes controller manager service using this command:

/usr/local/bin/kube-controller-manager --address=127.0.0.1 --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem --leader-elect=true --master=http://172.19.104.231:8080

show this log:

    E0905 23:04:10.866049   21126 leaderelection.go:359] Failed to update lock: Operation cannot be fulfilled on endpoints "kube-controller-manager": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/kube-system/kube-controller-manager, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 0c84a2ba-80f5-11e9-b3ce-00163e086f0c, UID in object meta: eec95e0e-65a1-4941-be87-d70f5f2429d7
E0905 23:04:12.197416   21510 leaderelection.go:359] Failed to update lock: Operation cannot be fulfilled on endpoints "kube-scheduler": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/kube-system/kube-scheduler, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 121bc661-80f5-11e9-b3ce-00163e086f0c, UID in object meta: 49e84916-589a-4da5-b78a-761a1fe78285

what should I do to fix this error? This is kubectl version:

[root@iZuf63refzweg1d9dh94t8Z ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:23:26Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
-- Dolphin
kubernetes

Similar Questions

Rancher Cluster Monitoring + Prometheus Operator?
Have multiple Urls for viewing logs in spinnaker
SEQ on Kubernetes not publicly visible
How to collect all ip's of pods by specific name filter
How to design readiness and livness probe for Angular application

1 Answer

9/12/2019

First of all thanks for providing additional info I requested. However it is still not clear how and what you configured else.

But I can see you decided to go through "the hard way". It is not possible to reproduce all the thing you configured but..

Some time ago I successfully passed cluster creation procedure following by widely known kelseyhightower/kubernetes-the-hard-way .

You can find there step-by-step instruction for configuring each part in k8s cluster.

Take a look on Configure the Kubernetes Controller Manager part under Bootstrapping the Kubernetes Control Plane document.

Configure the Kubernetes Controller Manager:

Move the kube-controller-manager kubeconfig into place:

sudo mv kube-controller-manager.kubeconfig /var/lib/kubernetes/

Create the kube-controller-manager.service systemd unit file:

cat <<EOF | sudo tee /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
  --address=0.0.0.0 \\
  --cluster-cidr=10.200.0.0/16 \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
  --leader-elect=true \\
  --root-ca-file=/var/lib/kubernetes/ca.pem \\
  --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \\
  --service-cluster-ip-range=10.32.0.0/24 \\
  --use-service-account-credentials=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

But again, this part wont work without other important pieces of this puzzle. Hope it will help you to achieve the expected result

-- VKR
Source: StackOverflow