Currently in my kubernetes-nodes job in Prometheus, The endpoint /api/v1/nodes/gk3-<cluster name>-default-pool-<something arbitrary>/proxy/metrics is being scraped
But the thing is I'm getting a 403 error which says GKEAutopilot authz: cluster scoped resource "nodes/proxy" is managed and access is denied when I try it manually on postman
How do I get around this on GKE Autopilot?
While the Autopilot docs don't mention the node proxy API specifically, this is in the limitations section:
Most external monitoring tools require access that is restricted. Solutions from several Google Cloud partners are available for use on Autopilot, however not all are supported, and custom monitoring tools cannot be installed on Autopilot clusters.
Given that port-forward and all other node-level access is restricted it seems likely this is not available. It's not clear that Autopilot even uses Kubelet at all and they probably aren't going to tell you.
End of year update:
This mostly works now. Autopilot has added support for things like cluster-scope objects and webhooks. You do need to reconfigure any install manifests to not touch the kube-system namespace as that is still locked down but you can most of this working if you hammer on it a bunch.
Created a firewall to allow ingress traffic to port 10250-10255 (kubelet)
$ gcloud compute firewall-rules create test-kubelet-ingress --allow tcp:10250-10255 --source-ranges="0.0.0.0/0"
Ran the following to:
### make sure the user can create nodes/proxy
$ kubectl config view
$ kubectl get all --all-namespaces
$ kubectl create clusterrolebinding autopilot-cluster-1 --clusterrole=k8-cluster-1 --user=infosys-khajashaik@premium-cloud-support.com
### checking
$ kubectl auth can-i create nodes/proxy
#> output
# Warning: resource 'nodes' is not namespace scoped
# yes
$ curl -k https://{NODE_PUBLIC_IP}:10250/run/kube-system/{POD_NAME}/netd -d "cmd=ls" --header "Authorization: Bearer $TOKEN" --insecure
TOKEN = <auto generated token in local kubeconfig>
NODE_PUBLIC_IP = <the public ip of the node>
POD_NAME = <netd pod name in the node>
So even though the user has permissions in the kube-apiserver, it is denied to create a "nodes/proxy" by kubelet.
If nodes/proxy is removed from the authz, it success creating a proxy
$ curl -k https://35.202.254.215:10250/run/kube-system/netd-ff5vr/netd -d "cmd=ls" --header "Authorization: Bearer $TOKEN" --insecure