K
Q

listing/creation of PV failing using kubernetes RBAC

3/10/2021

i have a service account which has access to one of the app namespace. I have created a cluster role and rolebinding and mapped it to the associated Service account in that namespace. Everything works as expected except the listing/creation of PV on the cluster level. Can some please help.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: dxf-clusterrole
rules:
  -
    apiGroups:
      - ""
      - apps
      - batch
      - extensions
      - policy
      - rbac.authorization.k8s.io
      - roles.rbac.authorization.k8s.io
      - authorization.k8s.io
    resources:
      - secrets
      - configmaps
      - deployments
      - endpoints
      - horizontalpodautoscalers
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
      - role
      - rolebindings

    verbs:
      - get
      - watch
      - list
      - create
      - delete
  - nonResourceURLs: ["*"]
    verbs:
      - get
      - watch
      - list



apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  creationTimestamp: null
  name: dxf-clusterrolebinding
  namespace: dxf-uat
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: dxf-clusterrole
subjects:
- kind: ServiceAccount
  name: dxf-deployer
  namespace: dxf-uat

User "system:serviceaccount:dxf-uat:dxf-deployer" cannot get resource "persistentvolumes" in API group "" at the cluster scope

-- cks cks
kubernetes
rbac

Similar Questions

How to install multiple version of Kubectl on Ubuntu?
Communication from Pod to Pod on same node inside kubernetes in GCP
Externally Access Hadoop HDFS deployed in Kubernetes
Kubernetes: Understanding resource requests/limits and scheduling
Connecting GKE Kube Master with ssh

1 Answer

3/10/2021

There are four Kubernetes objects: Role, ClusterRole, RoleBinding and ClusterRoleBinding, that we can use to configure needed RBAC rules. Role and RoleBinding are namespaced and ClusterRole and ClusterRoleBinding are cluster scoped resources.

As you can see in the RoleBinding and ClusterRoleBinding documentation:

A RoleBinding grants permissions within a specific namespace whereas a ClusterRoleBinding grants that access cluster-wide.

Your problem is with all cluster-scope resources such as PersistentVolumes, Nodes, Namespaces etc:

$ kubectl get nodes --as=system:serviceaccount:dxf-uat:dxf-deployer
Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:dxf-uat:dxf-deployer" cannot list resource "nodes" in API group "" at the cluster scope

$ kubectl get persistentvolumes -n dxf-uat --as=system:serviceaccount:dxf-uat:dxf-deployer
Error from server (Forbidden): persistentvolumes is forbidden: User "system:serviceaccount:dxf-uat:dxf-deployer" cannot list resource "persistentvolumes" in API group "" at the cluster scope

$ kubectl get namespaces --as=system:serviceaccount:dxf-uat:dxf-deployer
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:dxf-uat:dxf-deployer" cannot list resource "namespaces" in API group "" at the cluster scope

You need to create a ClusterRole with all cluster scoped resources you want to have access from the dxf-deployer ServiceAccount and then bind this ClusterRole to the dxf-deployer ServiceAccount using ClusterRoleBinding.

In the example below, I've granted permissions for dxf-deployer ServiceAccount to Nodes and PersistentVolumes:

$ cat cluster-scope-permissions.yml
# cluster-scope-permissions.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-scope-role
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - persistentvolumes
  verbs:
  - get
  - list
  - watch
  - create
  - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-scope-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-scope-role
subjects:
- kind: ServiceAccount
  name: dxf-deployer
  namespace: dxf-uat

Finally, we can check if it works as expected:

$ kubectl apply -f cluster-scope-permissions.yml
clusterrole.rbac.authorization.k8s.io/cluster-scope-role created
clusterrolebinding.rbac.authorization.k8s.io/cluster-scope-rolebinding created


$ kubectl get nodes --as=system:serviceaccount:dxf-uat:dxf-deployer
NAME                                       STATUS   ROLES    AGE     VERSION
node1                                      Ready    <none>   5h11m   v1.18.12-gke.1210
node2                                      Ready    <none>   5h11m   v1.18.12-gke.1210

$ kubectl get persistentvolumes -n dxf-uat --as=system:serviceaccount:dxf-uat:dxf-deployer
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   REASON   AGE
pvc-0ba2fd12-c883-45b8-b52d-a6c826a2775a   8Gi        RWO            Delete           Bound    default/my-jenkins   standard                131m
pvc-b4b7a4c8-c9ad-4e83-b1ee-663b3e4d938b   10Gi       RWO            Delete           Bound    default/debug-pvc    standard                5h12m
-- matt_j
Source: StackOverflow