10.9. PCI(e) Passthrough

PCI(e) passthrough is a mechanism to give a virtual machine control over a PCI device from the host. This can have some advantages over using virtualized hardware, for example lower latency, higher performance, or more features (e.g., offloading).

But, if you pass through a device to a virtual machine, you cannot use that device anymore on the host or in any other VM.

Since passthrough is a feature which also needs hardware support, there are some requirements to check and preparations to be done to make it work.

Once you ensured that your hardware supports passthrough, you will need to do some configuration to enable PCI(e) passthrough.

IOMMU. First, the IOMMU support has to be enabled in your BIOS/UEFI. Most often, that options is named IOMMU or VT-d, but check the manual for your motherboard for the exact option you need to enable.

Then, the IOMMU has to be activated on the kernel commandline Section 3.12.6, “Editing the Kernel Commandline”.

The command line parameters are:

  • for Intel CPUs:

     intel_iommu=on
  • for AMD CPUs it should be enabled automatically.

Kernel Modules. You have to make sure the following modules are loaded. This can be achieved by adding them to ‘/etc/modules

 vfio
 vfio_iommu_type1
 vfio_pci
 vfio_virqfd

After changing anything modules related, you need to refresh your initramfs. On Proxmox VE this can be done by executing:

# update-initramfs -u -k all

Finish Configuration. Finally reboot to bring the changes into effect and check that it is indeed enabled.

# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

should display that IOMMU, Directed I/O or Interrupt Remapping is enabled, depending on hardware and kernel the exact message can vary.

It is also important that the device(s) you want to pass through are in a separate IOMMU group. This can be checked with:

# find /sys/kernel/iommu_groups/ -type l

It is okay if the device is in an IOMMU group together with its functions (e.g. a GPU with the HDMI Audio device) or with its root port or PCI(e) bridge.

The most used variant of PCI(e) passthrough is to pass through a whole PCI(e) card, for example a GPU or a network card.

Another variant for passing through PCI(e) devices, is to use the hardware virtualization features of your devices, if available.

SR-IOV (Single-Root Input/Output Virtualization) enables a single device to provide multiple VF (Virtual Functions) to the system. Each of those VF can be used in a different VM, with full hardware features and also better performance and lower latency than software virtualized devices.

Currently, the most common use case for this are NICs (Network Interface Card) with SR-IOV support, which can provide multiple VFs per physical port. This allows using features such as checksum offloading, etc. to be used inside a VM, reducing the (host) CPU overhead.

Mediated devices are another method to reuse features and performance from physical hardware for virtualized hardware. These are found most common in virtualized GPU setups such as Intel’s GVT-g and NVIDIA’s vGPUs used in their GRID technology.

With this, a physical Card is able to create virtual cards, similar to SR-IOV. The difference is that mediated devices do not appear as PCI(e) devices in the host, and are such only suited for using in virtual machines.