System

uname -a output: Linux de-s1.bitinflow.com 5.7.0-0.bpo.2-amd64 #1 SMP Debian 5.7.10-1~bpo10+1 (2020-07-30) x86_64 GNU/Linux

Software Requirements

sudo apt-get install netcat-openbsd
git clone https://github.com/bitinflow/cloud.git /opt/cloud

Install the bitinflow Cloud Framework

bash /opt/cloud/cloud core:kickstart

Do you want to install the core? y
Do you want to install the systemd service? y

Enabling IOMMU feature

Edit Grub File

sudo nano /etc/default/grub

Match GRUB_CMDLINE_LINUX_DEFAULT with the following (on AMD CPUs only):

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1"

If you have Intel CPUs, use the follwing settings:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"

Update Grub

sudo update-grub

Verify after Reboot

dmesg |grep AMD-Vi

[    0.751574] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.757140] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.757234] pci 0000:00:00.2: AMD-Vi: Extended features (0x58f77ef22294ade):
[    0.757424] AMD-Vi: Interrupt remapping enabled
[    0.757514] AMD-Vi: Virtual APIC enabled
[    0.757602] AMD-Vi: X2APIC enabled
[    0.757777] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.768441] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

Identifiy your Hardware

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

...
IOMMU Group 16 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
IOMMU Group 16 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
IOMMU Group 16 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
IOMMU Group 16 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 16 02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 16 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 16 03:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 16 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)
IOMMU Group 16 05:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
...
IOMMU Group 17 06:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)
IOMMU Group 17 06:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)
...
IOMMU Group 18 07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 18 07:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
...

Selected Devices for Isolation

We are looking for the device id of the guest GPU for isolation. Keep in mind that the GPU usually comes combined with an audio device.

IOMMU Group 18 07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 18 07:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

We have to isolate 10de:13c2 and 10de:0fbb for our GPU-Id's.

Isolation of the guest GPU

For the next step only the GPU-Id's is needed.

Then Modify /etc/initramfs-tools/modules and add:

vfio vfio_iommu_type1 vfio_virqfd vfio_pci ids=10de:13c2,10de:0fbb

Modify /etc/modules aswell, and add:

vfio vfio_iommu_type1 vfio_pci ids=10de:13c2,10de:0fbb

In order to alter the load sequence in favour to vfio_pci before the nvidia driver, create a file in the modprobe.d folder via /etc/modprobe.d/nvidia.conf, and add:

softdep nouveau pre: vfio-pci 
softdep nvidia pre: vfio-pci 
softdep nvidia* pre: vfio-pci

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:13c2,10de:0fbb

sudo nano /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1

sudo update-initramfs -u -k all

-> reboot the system.

Verify the isolation

lspci -nnv

Find the line "Kernel driver in use" for the GPU and its audio part. It should state vfio-pci.

...
07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) (prog-if 00 [VGA controller])                             [110/3444]
        Subsystem: Gigabyte Technology Co., Ltd GM204 [GeForce GTX 970] [1458:366a]
        Flags: fast devsel, IRQ 100
        Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

07:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd GM204 High Definition Audio Controller [1458:366a]
        Flags: bus master, fast devsel, latency 0, IRQ 104
        Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
...