K
Q

Segregate pod traffic with SR-IOV

8/1/2021

I understand that, SR-IOV enables a physical NIC to be "seen" as multiple devices and used individually. (Refer here.) While the following example is from Charmed Kubernetes, it is representative of the general mechanism of using SR-IOV in pods.

My question is as follows. If there were another pod definition (similar to the one below), can I make that pod to use a particular "device" exclusively? Conversely, will the other pod definition share the "device" with the earlier defined pod? Or, is SR-IOV meant for multi-homing pods only?

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/networks: sriov
spec:
  containers:
  - name: ubuntu
    image: ubuntu:20.04
    command: ['sleep', '3600']
    resources:
      requests:
        intel.com/intel_sriov_netdevice: '1'
      limits:
        intel.com/intel_sriov_netdevice: '1'
-- cogitoergosum
dma
kubernetes
kubernetes-networking
kubernetes-pod
numa

Similar Questions

AWS EKS K8s cluster with Weave CNI plugin cannot have custom-metrics
Google Kubernetes is not auto scaling to 0
Best practice for automating test data ingest to MongoDB in kubernetes cluster
How to replace load balancer Docker container with no downtime in a simple way
Difference between GKE Stackdriver Monitoring and GKE usage metering with BigQuery

1 Answer

9/28/2021

See the SR-IOV CNI page, esp. the sections Parameters and Runtime Configuration.

A CNI plugin in general needs to handle MAC and IP address assignments for the pod interfaces that it manages. For the SR-IOV CNI, the MAC address of the pod interface, i.e., the source MAC address in emitted packets, is none other than the MAC address of the PCI VF in the NIC that the pod interface is bound to. So, the SR-IOV CNI allows the MAC address to be assigned in one of two ways:

  • In the SR-IOV CNI's config file, the cluster admin can specify the MAC address for a specific VF based on its PCI bus-device-function address. See Parameters.
  • In the pod metadata, the K8s API user can specify the network and MAC address as an annotation. See Runtime Configuration. The SR-IOV CNI will configure the chosen VF with this MAC, if it is a valid MAC.

can I make that pod to use a particular "device" exclusively?

Define the term "device." You can choose a MAC in the pod metadata that will get applied to whatever VF gets chosen. I don't think you can choose a VF, because that violates the principle of letting the orchestrator (Kubernetes) manage the resources in the cluster.

Conversely, will the other pod definition share the "device" with the earlier defined pod?

If "device" means a VF, it cannot be shared among multiple pods. It is passed through as a PCI device to a specific container in a specific pod. All the PCI interactions for that VF -- memory-mapped IO (for register reads/writes), DMA and Interrupts -- can only be mapped to one user space domain (a container in this context) at one time.

Or, is SR-IOV meant for multi-homing pods only?

A multi-homing pod means that the pod has multiple interfaces. Any pod that uses SR-IOV must be multi-homed, because you would first use the Multus CNI to create multiple interfaces, with the primary interface sitting on your default CNI and another interface on SR-IOV CNI.

-- SundarNadathur
Source: StackOverflow