Translation(s): none


Introduction

Here's a How-To on creating a VGA passthrough with QEMU (especially useful for Windows as guest). The required features are quite recent and may not work on all hardware and guests. Hopefully this How-To should save you some time to setup the whole.

Tested Hardware

The hardware listed below is for reference. Use it as a guide if you need to buy new hardware but keep in mind software is evolving and your results may vary.

Configuration Year tested CPU Motherboard Host video card Guest video card
#1 (Intel + AMD) 2014 Intel i7-4770S Gigabyte Z87N-WIFI Intel HD 4600 (integrated) Asus HD 6850 (AMD)
#2 (2x AMD) 2013 AMD FX-6200 Gigabyte 970A-UD3 ATI Radeon HD 6670 ATI Radeon HD 5770
#3 (Nvidia + AMD) 2016 AMD Phenom II X4 965 Gigabyte 990FXA-UD3 Nvidia GTX 650Ti ATI Radeon HD 5670

Prerequisites overview

You will need a Debian at least Wheezy. Your CPU should support virtualization and IOMMU (not supported by K variant of Intel CPUs). You need two distinct GPUs that can be used at the same time (Optimus cards won't work). The guest will output its display directly from the connected monitor (not visible from the host!), so you need two monitors or one with two inputs (one plugged into your host GPU, one into your guest GPU). You need a separate keyboard and mouse for the guest only (they are exclusively used by the guest) OR you could use device-sharing solutions like VNC or Synergy.

Hardware Setup

Configure your BIOS to make sure:

You can check directly from linux as follows:

egrep -q '^flags.*(svm|vmx)' /proc/cpuinfo && echo virtualization extensions available

Host OS Installation

You should have at least Debian Wheezy. Make sure you have a working Xorg server. You need QEMU with KVM (at least version 2.0.0):

aptitude install qemu-kvm

Optionally you can use a user interface (GUI) for managing your VMs with libvirt and virt-manager. If using libvirt / virt-manager, the tools might be restrictive - e.g.:

These limitations can be worked around by modifying the qemu commands which libvirt execute - e.g. by using a modified version of the method described here:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Technical_Notes/virt.html

...but it may be non-trivial to do-so.

We use only the qemu cli in the following.

The Debian Jessie kernel ships with CONFIG_VFIO_PCI_VGA=y, but for Wheezy one must compile the kernel himself to add this missing mandatory feature as it wass not in the debian kernel yet. You can read https://www.debian.org/releases/stable/i386/ch08s06.html.en if you don't know how.

On Intel platforms it is necessary to add intel_iommu=on on the kernel commandline (add in to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub and run update-grub).

Driver association

After that we need to assign the correct driver. This should be automatized into a script to be run at boot time. We need to use the VFIO driver for all pass-through devices so that the guest can manage them completely. At this point you should make sure the driver for the guest card is not loaded (eg: radeon, nouveau, …), you should blacklist them and reboot if necessary. In the case you have two cards with identical model you should use a PCI stub in the following (not describe here).

Find the PCI port, vendor and model of your card with lspci, here is an excerpt:

    # lspci -vnn

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739] (prog-if 00 [VGA controller])

01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:]

The card exposes the GPU and the HDMI soundcard. The GPU is at 0000:01:00.0 (pad with 0 on the left), vendor 1002, model 6739. The second needs to be unbound from its driver, in this case (using the port id):

    # echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind

Then we can load vfio and vfio_pci and assign all our devices to the vfio driver (using the vendor and model).

    # sudo modprobe vfio vfio_pci
    # echo 1002 6739 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
    # echo 1002 aa88 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

At this point you should be ready to start your VM with QEMU with special options described in the following part.

QEMU arguments (no GUI)

    export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0
    qemu-system-x86_64 \
        -enable-kvm -M q35 -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 \
        -bios /usr/share/qemu/bios.bin -vga none \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device piix4-ide,bus=pcie.0,id=piix4-ide \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$HOME/Asus.HD6850.1024.110616.rom \
        -device vfio-pci,host=01:00.1,bus=pcie.0 \
        -usb -usbdevice host:0603:00f2 -usbdevice host:046d:c01b \
        -soundhw ac97 \
        -drive file=$HOME/win7_rootfs.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \
        -drive file=$HOME/win7.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \
    ;

You should read the qemu manpage to ensure you understand what is happening. Replace the parameter of host= of the vfio-pci devices with your own card PCI ids (the GPU + the HDMI soundcard). You should provide the ROM for your GPU (romfile) but it could work without. Yours should be available at http://www.techpowerup.com/vgabios/ . We give the guest two USB devices (mouse+keyboard) using the vendor:model format (found with lsusb). Finally you should adapt for your disks and installation cdrom if relevant.

If you can boot up your guest OS then make sure your VGA device is visible. Then you can install the corresponding driver, for example in Windows 7, you can directly download them from the official website or using Windows Update.

You can optionally install the fedora virtio drivers and switch to virtio after rebooting and modifying the QEMU line. Fedora drivers: https://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/

Configuration #1 (Intel + AMD)

The measured performance was around 95% which is promising. Different 2D and 3D games were intensively tested (eg: Natural Selection 2). The platform is very stable. The GPU fan can be controlled with ?OverDrive (in Windows) and the sound + microphone both works.

Two very minor bugs occurs:

Tested versions

Configuration #2 (2x AMD)

Do not install the fglrx driver on the host. After I had the Windows 7 video passthrough working, I installed AMD's proprietary driver on the host to get a higher resolution on the host's console. It broke video passthrough. After testing it, I've concluded that just having the driver loaded causes a problem (even if I've assigned pci-stub to my passthrough card). The driver is causing my card to go into a busy state.

Simply issuing the following from a virtual console:

pkill gdm3    # after logging out, of course
rmmod fglrx
gdm3

fixed my problem. So take my advice: do not install the fglrx driver on the host.

Configuration #3 (Nvidia + AMD)

The ATI HD 5670 works flawlessly in 64-bit Windows 7 and Windows 10 and the performance very close to native. I used the latest AMD drivers (15.7.1 as of 12.03.16) and I have not encountered any problems.

The host uses the proprietary Nvidia drivers and isn't affected negatively by the passed through card.

Guest OS Installation

I know you're tempted to passthrough the video card right away. DON'T.

Start Virtual Machine Manager. Create a new virtual machine. Follow the wizard. DO NOT check off Customize configuration before install and assign the video card. Don't do it.

Install the OS (Windows 7). Once it is properly installed (the Windows installer will reboot a few times), shut it down. Go to the details screen for the guest OS in virtual manager. Use the "Add Hardware" button to add your video device. If you are using HDMI, don't forget to add the HDMI sound device. Do not remove the Video or Display VNC items.

Start your Windows 7 guest OS and verify that your device is present in the Device Manager. Then install the Windows driver for the graphics card you are passing through to your guest OS.

Reboot. Bob's your uncle. Or not. In my case I got the BSOD when booting the first 2 or 3 times. After rebooting the guest 3 or 4 times, it's worked consistently.

Your machine will start up displaying using VNC. Then the VNC image will freeze at the Windows logo and graphics will continue on the passthrough video card. This works well if you've also passed through a keyboard and mouse. Remember that you can't passthrough USB hubs and host controllers and expect their children to passthrough. Just passthrough the actual devices.

What if it doesn't work

There are a few options depending on your hardware that could make it work. Below are a few of them. You should read the Arch Linux forum (reference below) if it doesn't suffice.

Unsafe interrupts remapping

If your hardware doesn't support remapping of interruptions, you have to enable the unsafe assignments. Create /etc/modprobe.d/kvm_iommu.conf with:

options kvm allow_unsafe_assigned_interrupts=1

VGA arbiter + ACS override

If you have the same card multiple times you may need patches: VGA arbiter and ACS override. With kernel booting line: pcie_acs_override=downstream i915.enable_hd_vgaarb=1 .

Sound not working

Different soundcards were tried. It may not work for all configurations, here are my own results:

Related Resources