kubernetes V1.9, how to get kubelet.conf?

2/23/2018

i'm trying to build a Kubernetes cluster V1.9.3 with RBAC authz and admission-control options.

I've already do it with V1.6 and all works good, when my nodes starts the kubelet it use bootstrap file to get client certificate for kubelet. It get 5 files : kubelet.crt, kubelet.key, kubelet-client.key, kubelet-client.crt and the kubelet.conf.

In V1.9 i'am using the same configuration on all components, i just adapted them to the update 1.6->1.9. apiserver, scheduler and controller-manager starts without any issue.

When my nodes starts the kubelet, it use the bootstrap too and get kubelet.crt kubelet.key and kubelet-client.key, but neither kubelet-client.crt nor kubelet.conf file.

I see no issue in logs of all components, all seems OK, but my kubelet don't get it's config file

Here my apiserver config in 1.9:

--admission-control=NodeRestriction,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds 
--authorization-mode=Node,RBAC 
--secure-port=6443 
--bind-address=0.0.0.0 
--advertise-address=10.0.0.8 
--insecure-port=0 
--insecure-bind-address=127.0.0.1 
--client-ca-file=/etc/kubernetes/pki/ca.crt 
--etcd-servers=http://127.0.0.1:2379 
--service-account-key-file=/etc/kubernetes/pki/sa.pub 
--service-cluster-ip-range=172.30.100.0/24 
--tls-cert-file=/etc/kubernetes/pki/kube-apiserver.crt 
--tls-private-key-file=/etc/kubernetes/pki/kube-apiserver.key 
--enable-bootstrap-token-auth 
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key 
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname 
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt 
--requestheader-username-headers=X-Remote-User 
--requestheader-group-headers=X-Remote-Group 
--requestheader-allowed-names=front-proxy-client 
--requestheader-extra-headers-prefix=X-Remote-Extra-

Kubelet config in 1.9 :

--bootstrap-kubeconfig=/etc/kubernetes/kubeconfig
--kubeconfig=/etc/kubernetes/kubelet.conf   
--pod-manifest-path=/etc/kubernetes/manifests   
--allow-privileged=true   
--network-plugin=cni   
--cni-conf-dir=/etc/cni/net.d   
--cni-bin-dir=/opt/cni/bin   
--cluster-dns=172.30.0.10   
--cluster-domain=cluster.local   
--authorization-mode=Webhook   
--client-ca-file=/etc/kubernetes/pki/ca.crt   
--cadvisor-port=0   
--rotate-certificates=true   
--cert-dir=/var/lib/kubelet/pki  
--cgroup-driver=systemd

systemctl status -l kubelet :

Feb 23 17:24:43 kubenode1 systemd[1]: Started Kubernetes Kubelet.
Feb 23 17:24:43 kubenode1 systemd[1]: Starting Kubernetes Kubelet...
Feb 23 17:24:43 kubenode1 kubelet[12489]: I0223 17:24:43.664804   12489 feature_gate.go:226] feature gates: &{{} map[]}
Feb 23 17:24:43 kubenode1 kubelet[12489]: I0223 17:24:43.664956   12489 controller.go:114] kubelet config controller: starting controller
Feb 23 17:24:43 kubenode1 kubelet[12489]: I0223 17:24:43.664960   12489 controller.go:118] kubelet config controller: validating combination of defaults and flags
Feb 23 17:24:44 kubenode1 kubelet[12489]: W0223 17:24:44.078716   12489 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Feb 23 17:24:44 kubenode1 kubelet[12489]: I0223 17:24:44.082194   12489 server.go:182] Version: v1.9.3
Feb 23 17:24:44 kubenode1 kubelet[12489]: I0223 17:24:44.082228   12489 feature_gate.go:226] feature gates: &{{} map[]}
Feb 23 17:24:44 kubenode1 kubelet[12489]: I0223 17:24:44.082310   12489 plugins.go:101] No cloud provider specified.

ls /var/lib/kubelet/pki :

kubelet-client.key
kubelet.crt
kubelet.key

How can i force kubelet to get kubelet.conf file in v1.9 ? Someone have any idea ?

-- yoh852
kubelet
kubernetes

1 Answer

2/24/2018

I think I found the problem, the controller-manager doesn't auto approve the kubelet request anymore. I have to use the following commands to approve it :

$ kubectl -n kube-system get csr
NAME                                                   AGE       REQUESTOR                 CONDITION
node-csr-a1NZL_Efc2eDyrtSNZcIL1NxuHTvrsCO-ObfFFRUh8U   27s       system:bootstrap:61bb3c   Pending
node-csr-co5i3DRYeYOwCBff5XYA0rfSIqniy05VtdiVTmarQQA   5h        system:bootstrap:61bb3c   Approved,Issued
$ kubectl certificate approve node-csr-IoN3CNTJmJHGKUWso1OXLwd1H3KLOon4oVCz3uVYfqw
certificatesigningrequest "node-csr-a1NZL_Efc2eDyrtSNZcIL1NxuHTvrsCO-ObfFFRUh8U" approved

I followed this doc thinking that would enable the auto-approve, but it's seems not :/ https://kubernetes.io/docs/admin/kubelet-tls-bootstrapping/#kubectl-approval

-- yoh852
Source: StackOverflow