A client requires local kube cluster due to difficulty in accessing the cloud due to their location. Each server has a raid 1 set for OS and multiple TB drives for ceph fs. We would like to optimize the drives fully for ceph fs and build kube on top of it. However based on my research, the rook plugin for kube basically only provides PV support with ceph fs. Does this mean we must build a separate RAID set for the kube pods and other infrastructure and use ceph fs primarily for PV's?
If you check Kubernetes CephFS volume documentation
Caution: You must have your own Ceph server running with the share exported before you can use it.
So, yes, you can't use your OS filesystems out of the box for cephFS PV for Kubernetes Cluster.
Hope it helps!