When I follow the guide for Kubernetes block devices on Ceph RBD at https://docs.ceph.com/docs/master/rbd/rbd-kubernetes/ I get the following error at the user creation.
~# ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes' Error EINVAL: mon capability parse failed, stopped at 'pool=kubernetes' of 'profile rbd pool=kubernetes'
I can create the user if I remove the mgr capabilities. Does Kubernetes Ceph client needs mgr capabilities ?
If yes, how should I configured that user ?
With the above capabilities definition, I have the below error from the provisioner:
$ kubectl logs csi-rbdplugin-provisioner-6956bdfdf9-knpvt csi-provisioner
I0403 21:37:12.449780 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com
I0403 21:37:15.750596 1 controller.go:1199] provision "default/rbd-pvc" class "csi-rbd-sc": started
I0403 21:37:15.753508 1 controller.go:494] CreateVolumeRequest {Name:pvc-857dbdb6-fc82-40b0-b78c-d3ca675741b0 CapacityRange:required_bytes:1073741824 VolumeCapabilities:[mount:<fs_type:"ext4" mount_flags:"discard" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[adminId:admin clusterID:3ce42410-da82-4467-bc36-258e1f2217b1 csi.storage.k8s.io/node-stage-secret-name:csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace:default csi.storage.k8s.io/provisioner-secret-name:csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace:default monitors:10.250.20.21:6789,10.250.20.22:6789,10.250.20.23:6789 pool:kubernetes userId:kubernetes] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0403 21:37:15.753733 1 event.go:255] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"857dbdb6-fc82-40b0-b78c-d3ca675741b0", APIVersion:"v1", ResourceVersion:"6475509", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/rbd-pvc"
I0403 21:37:15.758339 1 connection.go:180] GRPC call: /csi.v1.Controller/CreateVolume
I0403 21:37:15.758356 1 connection.go:181] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-857dbdb6-fc82-40b0-b78c-d3ca675741b0","parameters":{"adminId":"admin","clusterID":"3ce42410-da82-4467-bc36-258e1f2217b1","monitors":"10.250.20.21:6789,10.250.20.22:6789,10.250.20.23:6789","pool":"kubernetes","userId":"kubernetes"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0403 21:37:16.736899 1 connection.go:183] GRPC response: {}
I0403 21:37:16.737220 1 connection.go:184] GRPC error: rpc error: code = Internal desc = failed to get IOContext: failed to get connection: connecting failed: rados: ret=1, Operation not permitted
I0403 21:37:16.737260 1 controller.go:1016] Final error received, removing PVC 857dbdb6-fc82-40b0-b78c-d3ca675741b0 from claims in progress
W0403 21:37:16.737273 1 controller.go:887] Retrying syncing claim "857dbdb6-fc82-40b0-b78c-d3ca675741b0", failure 60
E0403 21:37:16.737289 1 controller.go:910] error syncing claim "857dbdb6-fc82-40b0-b78c-d3ca675741b0": failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = failed to get IOContext: failed to get connection: connecting failed: rados: ret=1, Operation not permitted
I0403 21:37:16.737338 1 event.go:255] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"857dbdb6-fc82-40b0-b78c-d3ca675741b0", APIVersion:"v1", ResourceVersion:"6475509", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = failed to get IOContext: failed to get connection: connecting failed: rados: ret=1, Operation not permitted
I0403 21:37:17.486310 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-comThanks !