I'm looking to deploy a bare metal k8s cluster.
Typically when I deploy k8s clusters, I have two networks. Control Plane, and Nodes. However, in this cluster I'd like to leverage rook to present storage (ceph/nfs).
Most advice I get and articles I read say that systems like ceph need their own backend, isolated cluster network for replication etc - ceph reference docs. Moreover, a common datacenter practice is to have a separate network for NFS.
How are these requirements and practices adopted in a k8s world? Can the physical network just be flat, and the k8s SDN does all the heavy lifting here? Do I need to configure network policies and additional interfaces to provide physical segregation for my resources?
Ceph best practice is to have separate "cluster network" for replication/rebalancing and client-facing network (so called "public network") which is used by clients (like K8s nodes) to connect to Ceph. Ceph cluster network is totally different from K8s cluster network. Those are simply two different things. Ideally they should live on different NICs and switches/switchports.
If you have separate NICs towards Ceph cluster then you can create interfaces on K8s nodes to interact with Ceph's "public network" using those dedicated NICs. So there will be separate interfaces for K8s management/inter-pod traffic and separate interfaces for storage traffic.