Differences between revisions 21 and 22
Revision 21 as of 2014-10-13 12:06:39
Size: 10697
Editor: ?TimSmall
Comment: Add VFIO tips and tricks blog to related resources.
Revision 22 as of 2014-10-13 14:06:39
Size: 11522
Editor: ?TimSmall
Comment: Insert notes regarding use of libvirt / virt-manager instead of "raw" qemu
Deletions are marked like this. Additions are marked like this.
Line 53: Line 53:
Optionally you can use a user interface (GUI) for managing your VMs with libvirt and virt-manager. We use only the qemu cli in the following. Optionally you can use a user interface (GUI) for managing your VMs with libvirt and virt-manager. If using libvirt / virt-manager, the tools might be restrictive - e.g.:
 * The PCI geometry they create by default may violate guest OS graphics drivers' expectations by placing the GPU directly on the PCI root bus, or behind a PCI (rather than PCI Express bridge).
 * It may be impossible to disable the virtualised VGA card.
 * Creating virtual PCs based on qemu's newer 'Q35' chipset (if desired) requires patching the tools by hand: https://www.redhat.com/archives/virt-tools-list/2014-May/msg00001.html

These limitations can be worked around by modifying the qemu commands which libvirt execute - e.g. by using a modified version of the method described here:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Technical_Notes/virt.html

...but it may be non-trivial to do-so.

We use only the qemu cli in the following.

Translation(s): none


Introduction

Here's a How-To on creating a VGA passthrough with QEMU (especially useful for Windows as guest). The required features are quite recent and may not work on all hardware and guests. Hopefully this How-To should save you some time to setup the whole.

Tested Hardware

The hardware listed below is for reference. Use it as a guide if you need to buy new hardware but keep in mind software is evolving and your results may vary.

Configuration #1 (Intel + AMD) in 2014

  • CPU: Intel i7-4770S
  • Motherboard: Gigabyte Z87N-WIFI
  • Host Intel HD 4600 (integrated in CPU)
  • Guest video card: Asus HD 6850 (AMD)

Configuration #2 (2x AMD) in 2013

  • CPU: AMD FX-6200
  • Motherboard: Gigabyte 970A-UD3
  • Host video card: ATI Radeon HD 6670
  • Guest video card: ATI Radeon HD 5770

Prerequisites overview

You will need a Debian at least Wheezy. Your CPU should support virtualization and IOMMU (not supported by K variant of Intel CPUs). You need two distinct GPUs that can be used at the same time (Optimus cards won't work). The guest will output its display directly from the connected monitor (not visible from the host!), so you need two monitors or one with two inputs (one plugged into your host GPU, one into your guest GPU). You need a separate keyboard and mouse for the guest only (they are exclusively used by the guest) OR you could use device-sharing solutions like VNC or Synergy.

Hardware Setup

Configure your BIOS to make sure:

  • CPU virtualization support is enabled;
  • VT-d (Intel) or IOMMU (AMD) is enabled.

You can check directly from linux as follows:

egrep -q '^flags.*(svm|vmx)' /proc/cpuinfo && echo virtualization extensions available

Host OS Installation

You should have at least Debian Wheezy. Make sure you have a working Xorg server. You need QEMU with KVM (at least version 2.0.0):

aptitude install qemu-kvm

Optionally you can use a user interface (GUI) for managing your VMs with libvirt and virt-manager. If using libvirt / virt-manager, the tools might be restrictive - e.g.:

  • The PCI geometry they create by default may violate guest OS graphics drivers' expectations by placing the GPU directly on the PCI root bus, or behind a PCI (rather than PCI Express bridge).
  • It may be impossible to disable the virtualised VGA card.
  • Creating virtual PCs based on qemu's newer 'Q35' chipset (if desired) requires patching the tools by hand: https://www.redhat.com/archives/virt-tools-list/2014-May/msg00001.html

These limitations can be worked around by modifying the qemu commands which libvirt execute - e.g. by using a modified version of the method described here:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Technical_Notes/virt.html

...but it may be non-trivial to do-so.

We use only the qemu cli in the following.

As of today, one must compile the kernel himself to add one missing mandatory feature: CONFIG_VFIO_PCI_VGA=y as it's not in the debian kernel yet. You can read https://www.debian.org/releases/stable/i386/ch08s06.html.en if you don't know how.

Driver association

After that we need to assign the correct driver. This should be automatized into a script to be run at boot time. We need to use the VFIO driver for all pass-through devices so that the guest can manage them completely. At this point you should make sure the driver for the guest card is not loaded (eg: radeon, nouveau, …), you should blacklist them and reboot if necessary. In the case you have two cards with identical model you should use a PCI stub in the following (not describe here).

Find the PCI port, vendor and model of your card with lspci, here is an excerpt:

    # lspci -vnn

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739] (prog-if 00 [VGA controller])

01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:]

The card exposes the GPU and the HDMI soundcard. The GPU is at 0000:01:00.0 (pad with 0 on the left), vendor 1002, model 6739. The second needs to be unbound from its driver, in this case (using the port id):

    # echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind

Then we can load vfio and vfio_pci and assign all our devices to the vfio driver (using the vendor and model).

    # sudo modprobe vfio vfio_pci
    # echo 1002 6739 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
    # echo 1002 aa88 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

At this point you should be ready to start your VM with QEMU with special options described in the following part.

QEMU arguments (no GUI)

    export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0
    qemu-system-x86_64 \
        -enable-kvm -M q35 -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 \
        -bios /usr/share/qemu/bios.bin -vga none \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device piix4-ide,bus=pcie.0,id=piix4-ide \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$HOME/Asus.HD6850.1024.110616.rom \
        -device vfio-pci,host=01:00.1,bus=pcie.0 \
        -usb -usbdevice host:0603:00f2 -usbdevice host:046d:c01b \
        -soundhw ac97 \
        -drive file=$HOME/win7_rootfs.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \
        -drive file=$HOME/win7.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \
    ;

You should read the qemu manpage to ensure you understand what is happening. Replace the parameter of host= of the vfio-pci devices with your own card PCI ids (the GPU + the HDMI soundcard). You should provide the ROM for your GPU (romfile) but it could work without. Yours should be available at http://www.techpowerup.com/vgabios/ . We give the guest two USB devices (mouse+keyboard) using the vendor:model format (found with lsusb). Finally you should adapt for your disks and installation cdrom if relevant.

If you can boot up your guest OS then make sure your VGA device is visible. Then you can install the corresponding driver, for example in Windows 7, you can directly download them from the official website or using Windows Update.

You can optionally install the fedora virtio drivers and switch to virtio after rebooting and modifying the QEMU line. Fedora drivers: https://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/

Configuration #1 (Intel + AMD)

The measured performance was around 95% which is promising. Different 2D and 3D games were intensively tested (eg: Natural Selection 2). The platform is very stable. The GPU fan can be controlled with ?OverDrive (in Windows) and the sound + microphone both works.

Two very minor bugs occurs:

  • Each time the VM boots, the Linux display has color glitches (partial inversion). One only need to switch of TTY: go to one console TTY and switch to X.
  • Sound may glitch a bit in video playback (rarely happens).

Tested versions

  • Host OS: linux-image-3.14-1-amd64=3.14.12-1 (compiled myself with CONFIG_VFIO_PCI_VGA=y CONFIG_HZ_1000=y) or 3.15 trunk (patches: trunk + acs_override)
  • QEMU: qemu-kvm=2.0.0+dfsg-6+b1 or from git ab6d3749c4915cd5692633e321f7745dce06fe77
  • Host packages: xserver-xorg=1:7.7+7 xserver-xorg-video-intel=2:2.21.15-2+b1 libegl1-mesa-drivers=10.2.2-1 firmware-linux=0.43 firmware-linux-free=3.3
  • Guest OS: Windows 7 SP1 Ultimate (64 Bit)
  • Drivers: 14.10.1006-140417a-171099C and amd_13_9_win7_win8_64_dd_ccc_whql.exe

Configuration #2 (2x AMD)

Do not install the fglrx driver on the host. After I had the Windows 7 video passthrough working, I installed AMD's proprietary driver on the host to get a higher resolution on the host's console. It broke video passthrough. After testing it, I've concluded that just having the driver loaded causes a problem (even if I've assigned pci-stub to my passthrough card). The driver is causing my card to go into a busy state.

Simply issuing the following from a virtual console:

pkill gdm3    # after logging out, of course
rmmod fglrx
gdm3

fixed my problem. So take my advice: do not install the fglrx driver on the host.

Guest OS Installation

I know your tempted to passthrough the video card right away. DON'T.

Start Virtual Machine Manager. Create a new virtual machine. Follow the wizard. DO NOT check off Customize configuration before install and assign the video card. Don't do it.

Install the OS (Windows 7). Once it is properly installed (the Windows installer will reboot a few times), shut it down. Go to the details screen for the guest OS in virtual manager. Use the "Add Hardware" button to add your video device. If you are using HDMI, don't forget to add the HDMI sound device. Do not remove the Video or Display VNC items.

Start your Windows 7 guest OS and verify that your device is present in the Device Manager. Then install the Windows driver for the graphics card you are passing through to your guest OS.

Reboot. Bob's your uncle. Or not. In my case I got the BSOD when booting the first 2 or 3 times. After rebooting the guest 3 or 4 times, it's worked consistently.

Your machine will start up displaying using VNC. Then the VNC image will freeze at the Windows logo and graphics will continue on the passthrough video card. This works well if you've also passed through a keyboard and mouse. Remember that you can't passthrough USB hubs and host controllers and expect their children to passthrough. Just passthrough the actual devices.

What if it doesn't work

There are a few options depending on your hardware that could make it work. Below are a few of them. You should read the Arch Linux forum (reference below) if it doesn't suffice.

Unsafe interrupts remapping

If your hardware doesn't support remapping of interruptions, you have to enable the unsafe assignments. Create /etc/modprobe.d/kvm_iommu.conf with:

options kvm allow_unsafe_assigned_interrupts=1

VGA arbiter + ACS override

If you have the same card multiple times you may need patches: VGA arbiter and ACS override. With kernel booting line: pcie_acs_override=downstream i915.enable_hd_vgaarb=1 .

Sound not working

Different soundcards were tried. It may not work for all configurations, here are my own results:

Related Resources