I have defined a ClusterRole for Prometheus:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
labels:
k8s-app: prometheus
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- endpoints
- services
- nodes
- pods
verbs:
- get
- watch
- list
- nonResourceURLs:
- /metrics
- /api/*
verbs:
- get
Prometheus is able to access the API-Servers /metrics route:
https://10.0.1.104:443/metrics
https://10.0.2.112:443/metrics
But I get "server returned HTTP status 403 Forbidden" on
https://kubernetes.default.svc:443/api/v1/nodes/ip-10-0-0-219.eu-west-1.compute.internal/proxy/metrics
and
https://kubernetes.default.svc:443/api/v1/nodes/ip-10-0-0-219.eu-west-1.compute.internal/proxy/metrics/cadvisor
I thought I had this covered by
- nonResourceURLs:
- /api/*
What am I missing?
I tried this myself and yes nodes/proxy
is missing. (it works for me after adding it)
rules:
- apiGroups: [""]
resources:
- namespaces
- endpoints
- services
- nodes
- nodes/proxy <===
- pods
# From my K8s master
$ curl -k -H 'Authorization: Bearer <redacted>' https://localhost:6443/api/v1/nodes/ip-x-x-x-x.us-west-2.compute.internal/proxy/stats/summary
{
"node": {
"nodeName": "ip-x-x-x-x.us-west-2.compute.internal",
"systemContainers": [
{
"name": "kubelet",
"startTime": "2018-10-19T21:02:19Z",
"cpu": {
"time": "2018-11-09T23:51:15Z",
"usageNanoCores": 30779949,
"usageCoreNanoSeconds": 59446529195638
},
....
Removing it:
$ curl -k -H 'Authorization: Bearer <redacted>' https://localhost:6443/api/v1/nodes/ip-x-x-x-x.us-west-2.compute.internal/proxy/stats/summary
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "nodes \"ip-x-x-x-x.us-west-2.compute.internal\" is forbidden: User \"system:serviceaccount:default:prometheus-k8s\" cannot get resource \"nodes/proxy\" in API group \"\" at the cluster scope",
"reason": "Forbidden",
"details": {
"name": "ip-x-x-x-x.us-west-2.compute.internal",
"kind": "nodes"
},
"code": 403
}
For those two endpoints, the rules may be missing nodes/metrics
and nodes/proxy
for (sub)resources, and possibly the proxy
verb.
If acceptable from security standpoint, it will be much easier to assign the cluster-reader
role to the prometheus' service account.