I am having trouble enabling webhook authentication for the kubelet API. My cluster is deployed with kubeadm. This post is similar, but not the same issue
I can authenticate to my API server with a bearer token just fine:
curl -k https://localhost:6443/api --header "Authorization: Bearer $TOKEN"I cannot authenticate against the kubelet api with the same header. I have enabled the following on the API server:
--authorization-mode=Node,RBAC
--anonymous-auth=false
--runtime-config=authentication.k8s.io/v1beta1=true,authorization.k8s.io/v1beta1=trueThe following is enabled on the kubelet node(s) (via /var/lib/kubelet/config.yaml)
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30sDespite this, I get a "403 forbidden" when curling the /metrics endpoint on the kubelet. Something to note, I can perform the same API call against a cluster deployed with KOPS just fine. I am not sure what the difference is.
The 403 indicates you successfully authenticated (or you would have gotten a 401 error), the kubelet checked with the apiserver if you were authorized to access kubelet metrics (otherwise it would have just allowed it), it got a definitely response from the apiserver (otherwise you would have gotten a 500 error), and the apiserver indicated the authenticated user is not authorized to access kubelet metrics.
See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/#kubelet-authorization for details about what permission needs to be granted to access various endpoints on the kubelet's API. For metrics, the nodes/metrics resource in the "" apiGroup must be granted.