I would like to install Kubernetes Metrics Server and try the Metrics API by following this recipe (from Kubernetes Handbook). I currently have a Kubernetes 1.13 cluster that was installed with kubeadm.
The recipe's section Enable API Aggregation recommends changes several settings in /etc/kubernetes/manifests/kube-apiserver.yaml. The current settings are as follows:
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
--requestheader-allowed-names=front-proxy-client
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-UserThe suggested new settings are as follows:
--requestheader-client-ca-file=/etc/kubernetes/certs/proxy-ca.crt
--proxy-client-cert-file=/etc/kubernetes/certs/proxy.crt
--proxy-client-key-file=/etc/kubernetes/certs/proxy.key
--requestheader-allowed-names=aggregator
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-UserIf I install metrics-server without these changes its log contains errors like this:
unable to fully collect metrics: ... unable to fetch metrics from Kubelet ... x509: certificate signed by unknown authority
Where do these credentials come from and what do they entail? I currently do not have a directory /etc/kubernetes/certs.
UPDATE I've now tried adding the following at suitable places inside metrics-server-deployment.yaml, however the issue still persists (in the absence of --kubelet-insecure-tls):
command:
- /metrics-server
- --client-ca-file
- /etc/kubernetes/pki/ca.crt
volumeMounts:
- mountPath: /etc/kubernetes/pki/ca.crt
name: ca
readOnly: true
volumens:
- hostPath:
path: /etc/kubernetes/pki/ca.crt
type: File
name: ca
UPDATE Here is probably the reason why mounting the CA certificate into the container apparently did not help.
Take a look on to how to Manage TLS Certificates in a Cluster:
Every Kubernetes cluster has a cluster root Certificate Authority (CA). The CA is generally used by cluster components to validate the API server’s certificate, by the API server to validate kubelet client certificates, etc. To support this, the CA certificate bundle is distributed to every node in the cluster and is distributed as a secret attached to default service accounts.
And also PKI Certificates and Requirements:
Kubernetes requires PKI certificates for authentication over TLS. If you install Kubernetes with
kubeadm, the certificates that your cluster requires are automatically generated.
kubeadm, by default, create the Kubernetes certificates at /etc/kubernetes/pki/ directory.
It looks like the metrics-server is trying to validate the kubelet serving certs without having them be signed by the main Kubernetes CA. Installation tools like kubeadm may don't set up certificates properly.
This problem can also happen in the case of your server have changed names/addresses after the Kubernetes installation, which causes a mismatch of the apiserver.crt Subject Alternative Name and your current names/addresses. Check it with:
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep DNSThe fastest/easy way to overcome this error is by using the --kubelet-insecure-tls flag for metrics-server. Something like this:
# metrics-server-deployment.yaml
[...]
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.1
command:
- /metrics-server
- --kubelet-insecure-tlsNote that this implies security concerns. If you are running for tests, ok. But for production, the best approach is to identify and fix the certificate issues (Take a look at this metrics-server issue for more information: #146)