Service account fails to delete resources, even though it should have rights to do so

5/31/2019

I have a service account monitoring:prometheus-operator-operator with a clusterrolebinding to to this clusterrole:

Name:         prometheus-operator-operator
Labels:       app=prometheus-operator-operator
              chart=prometheus-operator-5.7.0
              heritage=Tiller
              release=prometheus-operator
Annotations:  <none>
PolicyRule:
  Resources                                       Non-Resource URLs  Resource Names  Verbs
  ---------                                       -----------------  --------------  -----
  configmaps                                      []                 []              [*]
  secrets                                         []                 []              [*]
  customresourcedefinitions.apiextensions.k8s.io  []                 []              [*]
  statefulsets.apps                               []                 []              [*]
  alertmanagers.monitoring.coreos.com/finalizers  []                 []              [*]
  alertmanagers.monitoring.coreos.com             []                 []              [*]
  prometheuses.monitoring.coreos.com/finalizers   []                 []              [*]
  prometheuses.monitoring.coreos.com              []                 []              [*]
  prometheusrules.monitoring.coreos.com           []                 []              [*]
  servicemonitors.monitoring.coreos.com           []                 []              [*]
  endpoints                                       []                 []              [get create update]
  services                                        []                 []              [get create update]
  namespaces                                      []                 []              [get list watch]
  pods                                            []                 []              [list delete]
  nodes                                           []                 []              [list watch]

Now, I am attempting to run this

curl -ik -X DELETE \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  https://kubernetes.default.svc/apis/monitoring.coreos.com/v1/monitoring/prometheusrules/zalenium

from within a pod in the cluster to delete a PrometheusRule.

My request however is not successful and being rejected with a 403:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "monitoring.monitoring.coreos.com \"prometheusrules\" is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator-operator\" cannot delete resource \"monitoring/zalenium\" in API group \"monitoring.coreos.com\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "name": "prometheusrules",
    "group": "monitoring.coreos.com",
    "kind": "monitoring"
  },
  "code": 403
}

Am I wrong believing that the service account in my monitoring namespace should be able to delete PrometheusRule on the cluster level?

To me everything looks correct and I don't understand why I get a Forbidden response.

-- Moritz Schmitz v. Hülst
kubernetes
rbac

1 Answer

5/31/2019

you forgot to put namespace in the URI

curl -ik -X DELETE \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  https://kubernetes.default.svc/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules/zalenium

with the following command you can verify the if you are allowed to take action X with the resoucres Y

kubectl auth can-i delete prometheusrules --as system:serviceaccount:monitoring:prometheus-operator-operator -n monitoring

With the -v flag you can increase the verbosity of the request which also provides a request in curl form.

-- Suresh Vishnoi
Source: StackOverflow