Differences between revisions 16 and 17
Revision 16 as of 2013-02-09 09:48:48
Size: 4804
Editor: GeoffSimmons
Comment: InterWiki.
Revision 17 as of 2014-07-16 18:26:30
Size: 10748
Editor: ?d3lxa
Comment:
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
Here's a How To on creating a VGA passthrough for a Windows 7 guest. I spent a lot of time trying to get it to work. First with Xen: I couldn't get a straightforward HVM going even without a passthrough, and starting a guest would sometimes reboot my host machine! Then I tried KVM on Debian Squeeze: at least I could install Windows 7 and access it with a VNC session. I could not get the passthrough to work.

Eventually I tried Wheezy: and it almost works out of the box. I'll give you the steps to point you in the right direction, so you don't have to repeat my mistakes. I can't guarantee that this will work for your particular setup, but I hope that if it doesn't, you'll be far enough along that it won't take too much more effort to get it going.

My understanding is that some graphics cards will not work. Drivers may be written, and hardware designed, with assumptions in mind that aren't the case under virtualization and passthrough. You must also have support for hardware virtualization on your CPU and your '''motherboard'''.

VGA passthrough is in it's infancy. There isn't a lot of support available. It can be tricky to find useful resources and help online. With that in mind, I thought I'd write this.


= My Hardware =

You do not need to have my exact hardware, but it may be helpful to know that this worked if you have similar hardware or decide to run out and buy new hardware.
Here's a How-To on creating a VGA passthrough with QEMU (especially useful for Windows as guest).
The required features are quite recent and may not work on all hardware and guests.
Hopefully this How-To should save you some time to setup the whole.

= Tested Hardware =

The hardware listed below is for reference. Use it as a guide if you need to buy new hardware but keep in mind software and results may vary.

Configuration #1 (Intel + AMD) in 2014
 * CPU: Intel i7-4770S
 * Motherboard: Gigabyte Z87N-WIFI
 * Host Intel HD 4600 (integrated in CPU)
 * Guest video card: Asus HD 6850 (AMD)

Configuration #2 (2x AMD) in 2013
Line 26: Line 28:
= Prerequisites overview =

You will need a Debian at least Wheezy. You need two distinct GPUs that can be used at the same time (Optimus cards '''won't work'''). The guest will output its display directly from the connected monitor ('''not visible''' from the host!), so you need two monitors or one with two inputs (one plugged into your host GPU, one into your guest GPU). You need a separate keyboard and mouse for the guest only (they are exclusively used by the guest).
Line 28: Line 34:
Restart your machine and enter the BIOS. Make sure virtualization support is enabled (this is for the CPU). Also, make sure that VT-d (Intel) or IOMMU (AMD) is enabled (for the motherboard).

If you can, verify that your CPU supports virtualization. If you already have Linux installed or a Live CD, check support as follows.
Configure your BIOS to make sure:
 * CPU virtualization support is enabled
 * VT-d (Intel) or IOMMU (AMD) is enabled.

You can check directly from linux as follows:
Line 39: Line 47:
Install Debian Wheezy. Make sure the desktop environment is installed. After the installation is complete and you've rebooted, install KVM, libvirt and virt-manager (a user interface for creating, starting and managing virtual machines).

{{{
aptitude install qemu-kvm libvirt-bin virt-manager
}}}

Here's an '''important part:''' Make a file in /etc/modprobe.d (call it kvm_iommu.conf, call it whatever you want) with the following in it.

{{{
options kvm allow_unsafe_assigned_interrupts=1
}}}
You should have at least Debian Wheezy. Make sure you have a working Xorg server. You need QEMU with KVM (at least version 2.0.0):

{{{
aptitude install qemu-kvm
}}}

Optionally you can use a user interface (GUI) for managing for VMs with libvirt and virt-manager.

As of today, one must compile the kernel with a missing mandatory feature: '''CONFIG_VFIO_PCI_VGA=y''' it's not in the debian kernel yet.

Each device may have at most one driver associated, for the GPU it may be radeon, nouveau for example.
We need to use the VFIO driver to pass them unmanaged directly to the guest.
At this point you should make sure the drive for the guest card is not loaded, you can blacklist it and reboot if necessary.
If you have two identical model card you should use a PCI stub in the following.

Find the PCI port, vendor and model of your card with lspci, here is an excerpt:

{{{
    # lspci -vnn

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739] (prog-if 00 [VGA controller])
 Subsystem: ASUSTeK Computer Inc. EAH6850 [Radeon HD 6850] [1043:03b4]
 Flags: fast devsel, IRQ 16
 Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M]
 Memory at d0020000 (64-bit, non-prefetchable) [disabled] [size=128K]
 I/O ports at e000 [disabled] [size=256]
 Expansion ROM at d0000000 [disabled] [size=128K]
 Capabilities: <access denied>

01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:]
 Subsystem: ASUSTeK Computer Inc. Device [1043:aa88]
 Flags: fast devsel, IRQ 17
 Memory at d0040000 (64-bit, non-prefetchable) [disabled] [size=16K]
 Capabilities: <access denied>
 Kernel driver in use: snd_hda_intel
}}}

The card exposes the GPU and the HDMI soundcard. The GPU is at 0000:01:00.0 (pad with 0 on the left), vendor 1002, model 6739.
The second needs to be unbound from its driver, in this case (using the port id):

{{{
    # echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind
}}}

Then we can load vfio and vfio_pci and assign all our devices to the vfio driver (using the vendor and model).

{{{
    # sudo modprobe vfio vfio_pci
    # echo 1002 6739 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
    # echo 1002 aa88 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
}}}

At this point you should be ready to start your VM with QEMU with special options described in the following part.

= QEMU arguments (no GUI) =

{{{
    export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0
    qemu-system-x86_64 \
        -enable-kvm -M q35 -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 \
        -bios /usr/share/qemu/bios.bin -vga none \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device piix4-ide,bus=pcie.0,id=piix4-ide \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$HOME/Asus.HD6850.1024.110616.rom \
        -device vfio-pci,host=01:00.1,bus=pcie.0 \
        -usb -usbdevice host:0603:00f2 -usbdevice host:046d:c01b \
        -soundhw ac97 \
        -drive file=$HOME/win7_rootfs.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \
        -drive file=$HOME/win7.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \
    ;
}}}

You should read the qemu manpage to ensure you understand what is happening.
Replace the parameter of host= of the vfio-pci devices with your own card PCI ids (the GPU + the HDMI soundcard).
You should provide the ROM for your GPU (romfile) but it could work without. Yours should be available at http://www.techpowerup.com/vgabios/ .
We give the guest two USB devices (mouse+keyboard) using the vendor:model format (found with lsusb).
Finally you should adapt for your disks and installation cdrom if relevant.

If you can boot up your guest OS then make sure your VGA device is visible. Then you can install the corresponding driver, for example in Windows 7, you can directly download them from the official website or using Windows Update.

You can optionally install the fedora virtio drivers and switch to virtio after rebooting and '''modifying''' the QEMU line. Fedora drivers: https://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/

= Configuration #1 (Intel + AMD) =

The measured performance was around 95% which is promising. Different 2D and 3D games were intensively tested (eg: Natural Selection 2). The platform is very stable. The GPU fan can be controlled with OverDrive (in Windows) and the sound + microphone both works.

Two very minor bugs occurs:
    * Each time the VM boots, the Linux display has color glitches (partial inversion). One only need to switch of TTY: go to one console TTY and switch to X.
    * Sound may glitch a bit in video playback (rarely happens).

== Tested versions ==
 * Host OS: linux-image-3.14-1-amd64=3.14.12-1 (compiled myself with CONFIG_VFIO_PCI_VGA=y CONFIG_HZ_1000=y) or 3.15 trunk (patches: trunk + acs_override)
 * QEMU: qemu-kvm=2.0.0+dfsg-6+b1 or from git ab6d3749c4915cd5692633e321f7745dce06fe77
 * Host packages: xserver-xorg=1:7.7+7 xserver-xorg-video-intel=2:2.21.15-2+b1 libegl1-mesa-drivers=10.2.2-1 firmware-linux=0.43 firmware-linux-free=3.3
 * Guest OS: Windows 7 SP1 Ultimate (64 Bit)
 * Drivers: 14.10.1006-140417a-171099C and amd_13_9_win7_win8_64_dd_ccc_whql.exe

= Configuration #2 (2x AMD) =
Line 64: Line 159:
= Guest OS Installation = == Guest OS Installation ==
Line 78: Line 173:
= What if it doesn't work =
There are a few options depending on your hardware that could make it work. Below are a few of them. You should read the Arch Linux forum (reference below) if it doesn't suffice.

== Unsafe interrupts remapping ==
If your hardware doesn't support remapping of interruptions, you have to enable the unsafe assignments. Create /etc/modprobe.d/kvm_iommu.conf with:

{{{
options kvm allow_unsafe_assigned_interrupts=1
}}}


== VGA arbiter + ACS override ==
If you have the same card multiple times you may need patches: VGA arbiter and ACS override. With kernel booting line: pcie_acs_override=downstream i915.enable_hd_vgaarb=1 .

== Sound not working ==
Different soundcards were tried. It may not work for all configurations, here are my own results:

    * -soundhw ac97
        nearly perfect (some glitches)
        driver: http://www.realtek.com.tw/downloads/downloadsCheck.aspx?Langid=1&PNid=14&PFid=23&Level=4&Conn=3&DownTypeID=3&GetDown=false
    * -device usb-audio and QEMU_AUDIO_TIMER_PERIOD=10
        better but high CPU, may glitch under heavy load (or after closing program)
    * device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
        audio glitches (even with timer period to 0)
Line 80: Line 200:
 * [[https://bbs.archlinux.org/viewtopic.php?id=162768|Arch Linux Forum (very detailed)]]

Translation(s): none


Introduction

Here's a How-To on creating a VGA passthrough with QEMU (especially useful for Windows as guest). The required features are quite recent and may not work on all hardware and guests. Hopefully this How-To should save you some time to setup the whole.

Tested Hardware

The hardware listed below is for reference. Use it as a guide if you need to buy new hardware but keep in mind software and results may vary.

Configuration #1 (Intel + AMD) in 2014

  • CPU: Intel i7-4770S
  • Motherboard: Gigabyte Z87N-WIFI
  • Host Intel HD 4600 (integrated in CPU)
  • Guest video card: Asus HD 6850 (AMD)

Configuration #2 (2x AMD) in 2013

  • CPU: AMD FX-6200
  • Motherboard: Gigabyte 970A-UD3
  • Host video card: ATI Radeon HD 6670
  • Guest video card: ATI Radeon HD 5770

Prerequisites overview

You will need a Debian at least Wheezy. You need two distinct GPUs that can be used at the same time (Optimus cards won't work). The guest will output its display directly from the connected monitor (not visible from the host!), so you need two monitors or one with two inputs (one plugged into your host GPU, one into your guest GPU). You need a separate keyboard and mouse for the guest only (they are exclusively used by the guest).

Hardware Setup

Configure your BIOS to make sure:

  • CPU virtualization support is enabled
  • VT-d (Intel) or IOMMU (AMD) is enabled.

You can check directly from linux as follows:

egrep -q '^flags.*(svm|vmx)' /proc/cpuinfo && echo virtualization extensions available

Host OS Installation

You should have at least Debian Wheezy. Make sure you have a working Xorg server. You need QEMU with KVM (at least version 2.0.0):

aptitude install qemu-kvm

Optionally you can use a user interface (GUI) for managing for VMs with libvirt and virt-manager.

As of today, one must compile the kernel with a missing mandatory feature: CONFIG_VFIO_PCI_VGA=y it's not in the debian kernel yet.

Each device may have at most one driver associated, for the GPU it may be radeon, nouveau for example. We need to use the VFIO driver to pass them unmanaged directly to the guest. At this point you should make sure the drive for the guest card is not loaded, you can blacklist it and reboot if necessary. If you have two identical model card you should use a PCI stub in the following.

Find the PCI port, vendor and model of your card with lspci, here is an excerpt:

    # lspci -vnn

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739] (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. EAH6850 [Radeon HD 6850] [1043:03b4]
        Flags: fast devsel, IRQ 16
        Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Memory at d0020000 (64-bit, non-prefetchable) [disabled] [size=128K]
        I/O ports at e000 [disabled] [size=256]
        Expansion ROM at d0000000 [disabled] [size=128K]
        Capabilities: <access denied>

01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:]
        Subsystem: ASUSTeK Computer Inc. Device [1043:aa88]
        Flags: fast devsel, IRQ 17
        Memory at d0040000 (64-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: snd_hda_intel

The card exposes the GPU and the HDMI soundcard. The GPU is at 0000:01:00.0 (pad with 0 on the left), vendor 1002, model 6739. The second needs to be unbound from its driver, in this case (using the port id):

    # echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind

Then we can load vfio and vfio_pci and assign all our devices to the vfio driver (using the vendor and model).

    # sudo modprobe vfio vfio_pci
    # echo 1002 6739 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
    # echo 1002 aa88 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

At this point you should be ready to start your VM with QEMU with special options described in the following part.

QEMU arguments (no GUI)

    export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0
    qemu-system-x86_64 \
        -enable-kvm -M q35 -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 \
        -bios /usr/share/qemu/bios.bin -vga none \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device piix4-ide,bus=pcie.0,id=piix4-ide \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$HOME/Asus.HD6850.1024.110616.rom \
        -device vfio-pci,host=01:00.1,bus=pcie.0 \
        -usb -usbdevice host:0603:00f2 -usbdevice host:046d:c01b \
        -soundhw ac97 \
        -drive file=$HOME/win7_rootfs.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \
        -drive file=$HOME/win7.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \
    ;

You should read the qemu manpage to ensure you understand what is happening. Replace the parameter of host= of the vfio-pci devices with your own card PCI ids (the GPU + the HDMI soundcard). You should provide the ROM for your GPU (romfile) but it could work without. Yours should be available at http://www.techpowerup.com/vgabios/ . We give the guest two USB devices (mouse+keyboard) using the vendor:model format (found with lsusb). Finally you should adapt for your disks and installation cdrom if relevant.

If you can boot up your guest OS then make sure your VGA device is visible. Then you can install the corresponding driver, for example in Windows 7, you can directly download them from the official website or using Windows Update.

You can optionally install the fedora virtio drivers and switch to virtio after rebooting and modifying the QEMU line. Fedora drivers: https://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/

Configuration #1 (Intel + AMD)

The measured performance was around 95% which is promising. Different 2D and 3D games were intensively tested (eg: Natural Selection 2). The platform is very stable. The GPU fan can be controlled with ?OverDrive (in Windows) and the sound + microphone both works.

Two very minor bugs occurs:

  • Each time the VM boots, the Linux display has color glitches (partial inversion). One only need to switch of TTY: go to one console TTY and switch to X.
  • Sound may glitch a bit in video playback (rarely happens).

Tested versions

  • Host OS: linux-image-3.14-1-amd64=3.14.12-1 (compiled myself with CONFIG_VFIO_PCI_VGA=y CONFIG_HZ_1000=y) or 3.15 trunk (patches: trunk + acs_override)
  • QEMU: qemu-kvm=2.0.0+dfsg-6+b1 or from git ab6d3749c4915cd5692633e321f7745dce06fe77
  • Host packages: xserver-xorg=1:7.7+7 xserver-xorg-video-intel=2:2.21.15-2+b1 libegl1-mesa-drivers=10.2.2-1 firmware-linux=0.43 firmware-linux-free=3.3
  • Guest OS: Windows 7 SP1 Ultimate (64 Bit)
  • Drivers: 14.10.1006-140417a-171099C and amd_13_9_win7_win8_64_dd_ccc_whql.exe

Configuration #2 (2x AMD)

Do not install the fglrx driver on the host. After I had the Windows 7 video passthrough working, I installed AMD's proprietary driver on the host to get a higher resolution on the host's console. It broke video passthrough. After testing it, I've concluded that just having the driver loaded causes a problem (even if I've assigned pci-stub to my passthrough card). The driver is causing my card to go into a busy state.

Simply issuing the following from a virtual console:

pkill gdm3    # after logging out, of course
rmmod fglrx
gdm3

fixed my problem. So take my advice: do not install the fglrx driver on the host.

Guest OS Installation

I know your tempted to passthrough the video card right away. DON'T.

Start Virtual Machine Manager. Create a new virtual machine. Follow the wizard. DO NOT check off Customize configuration before install and assign the video card. Don't do it.

Install the OS (Windows 7). Once it is properly installed (the Windows installer will reboot a few times), shut it down. Go to the details screen for the guest OS in virtual manager. Use the "Add Hardware" button to add your video device. If you are using HDMI, don't forget to add the HDMI sound device. Do not remove the Video or Display VNC items.

Start your Windows 7 guest OS and verify that your device is present in the Device Manager. Then install the Windows driver for the graphics card you are passing through to your guest OS.

Reboot. Bob's your uncle. Or not. In my case I got the BSOD when booting the first 2 or 3 times. After rebooting the guest 3 or 4 times, it's worked consistently.

Your machine will start up displaying using VNC. Then the VNC image will freeze at the Windows logo and graphics will continue on the passthrough video card. This works well if you've also passed through a keyboard and mouse. Remember that you can't passthrough USB hubs and host controllers and expect their children to passthrough. Just passthrough the actual devices.

What if it doesn't work

There are a few options depending on your hardware that could make it work. Below are a few of them. You should read the Arch Linux forum (reference below) if it doesn't suffice.

Unsafe interrupts remapping

If your hardware doesn't support remapping of interruptions, you have to enable the unsafe assignments. Create /etc/modprobe.d/kvm_iommu.conf with:

options kvm allow_unsafe_assigned_interrupts=1

VGA arbiter + ACS override

If you have the same card multiple times you may need patches: VGA arbiter and ACS override. With kernel booting line: pcie_acs_override=downstream i915.enable_hd_vgaarb=1 .

Sound not working

Different soundcards were tried. It may not work for all configurations, here are my own results:

Related Resources