Translation(s): none
Migration of VMs between Xen and KVM - back and forth - the simple way
I am writing this in English because I spent one whole day by googling and I found no info about this even being possible. So I spent another day by frustrating experiments with no results, only to find very simple way which made me happy.
Contents
What we want to achieve
Migration of VMs between two Xen and KVM machines.
The migration does not have to be online, but it must be quick and easy.
The migration is possible because XEN is great and flexible system, allowing to customize all VMs in a way that they can be started in both XEN and KVM with no intermediate steps, so after the "one-time" modifications described below, no other VM conversions or modifications are necessary for the migrations.
The Xen is Debian 5 Lenny, KVM is Ubuntu 10.04 (~Debian Squeeze).
The XEN VM guests were created by the xen-create-image tool and we have too many of them, some with Debian 4, some with Debian 5, some with Debian 6. If all of these should be reinstalled from scratch, that would take month of work!
I do not guarantee that this will work in XEN 4 (Debian Squeeze XEN host), because I can't try it currently.
Why I need this
Because:
I need XEN for its easy PCI and USB passthrough on older HW without the VT-d support (see http://wiki.xensource.com/xenwiki/XenPCIpassthrough). However, KVM can't do PCI or USB passtrhrough on anything older than the i5500 chipset(see http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM).
- I've been using XEN for more than 5 years and I love it. It is rock-stable, it never failed. As for KVM, we have started using it year ago, it seems to work, but this is all I can say about it now.
How to do it
First of all, setup and start the DRBD devices, which will replicate your VM disks from the XEN to the KVM physical host machines.
Then, you must customize the XEN VM guests the following way:
/etc/inittab in all VM guest
You must comment out the following line, which is created by xen-create-image and directs console to the hvc0, which does not work in KVM:
#1:2345:respawn:/sbin/getty 38400 hvc0
Instead, place the standard tty1 console definition there:
1:2345:respawn:/sbin/getty 38400 tty1
Now, the console will not work in XEN. For enabling XEN console, you can place another line:
co:23:respawn:/sbin/getty -L hvc0 9600 vt102
Or you can redirect the hvc0 console to the tty1 by placing additional line in the VM guest definition file (e.g. /etc/xen/auto/logger.cfg)
extra = "console=hvc0 xencons=tty"
Modify VM guest definition files
You must rename your disks to sda/sdb/sdc... in the guest definition files (/etc/xen/auto/*.cfg).
By default, xen-create-image generates disk names with partition numbers in the form (hd|sd|xvd)[a-z][0-9], e.g. hda1, sda1 or xvda1
The problem is, that
- KVM can only handle disk names without partition numbers:
- * vd[a-z] when you use the virtio type
- * sda[a-z] when you use the "IDE" type
in next step, while still running the VM guests in XEN, we will install kernel and grub in each VM guest, but grub/grub2 in Lenny and Squeeze (grub2 versions < 1.99) have a bug (#601974) which does not allow installing it into a partition which is not part of a whole disk.
Therefore the only possible way forward is to use the sda/sdb/sdc... disk names in XEN.
Fortunatelly, XEN is happy with sda/sdb/sdc disk names so you can modify the guest definition files to like this:
#
# Configuration file for the Xen instance logger, created
# by xen-tools 3.9 on Wed May 19 02:56:03 2010.
#
#
# Kernel + memory size
#
kernel = '/boot/vmlinuz-2.6.26-2-xen-amd64'
ramdisk = '/boot/initrd.img-2.6.26-2-xen-amd64'
memory = '2048'
#
# Disk device(s).
#
root = '/dev/sda ro'
disk = [
'phy:/dev/vg_main/logger-swap,sdc,w',
'phy:/dev/drbd5,sda,w',
'phy:/dev/drbd6,sdb,w',
]
cpus="0-7"
vcpus=8
extra="clocksource=jiffies"
#for standard /etc/inittab (compatible with KVM):
extra = "console=hvc0 xencons=tty"
#
# Hostname
#
name = 'logger'
#
# Networking
#
dhcp = 'dhcp'
vif = [ 'mac=00:16:3E:37:D0:70' ]
#
# Behaviour
#
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
Change /etc/fstab in each guest VM
Before you restart (re-create) the guest VM, log into it and modify its /etc/fstab mount points to comply with your new disk names !
Install grub and kernel in each VM guest
Install Grub - it should be happy now with your /dev/sda root disk:
apt-get install grub2
Install the proper kernel file - e.g., for Debian Squeeze VM guest:
apt-get install linux-image-2.6.32-5-amd64
Make sure that the /boot/grub/device.map contains this line:
(hd0) /dev/sda
and then run
update-grub
which should output something like
Updating /boot/grub/grub.cfg ... Found linux image: /boot/vmlinuz-2.6.26-2-amd64 Found initrd image: /boot/initrd.img-2.6.26-2-amd64 done
Re-create and test the guest VM in XEN
Either
poweroff
in guest or
xm shutdown logger
in host, and then
xm create -c /etc/xen/auto/logger.cfg
Create new guest defintion file for KVM
On the KVM machine, thanks to DRBD, you already have the guest partitions ready, so the last step is to create the proper guest definitions, e.g. /root/logger.xml :
<domain type='kvm'>
<name>logger</name>
<uuid>30ea508e-7ed4-d590-3f63-0ea9d22e2545</uuid>
<memory>2097152</memory>
<currentMemory>2097152</currentMemory>
<vcpu>2</vcpu>
<os>
<type arch='x86_64' machine='pc-0.12'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/drbd5'/>
<target dev='sda' bus='ide'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/drbd6'/>
<target dev='sdb' bus='ide'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/vg_main/logger-swap'/>
<target dev='sdc' bus='ide'/>
</disk>
<interface type='bridge'>
<mac address='00:16:3E:37:D0:70'/>
<source bridge='br0'/>
<model type='virtio'/>
</interface>
<console type='pty'>
<target port='0'/>
</console>
<console type='pty'>
<target port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'/>
<video>
<model type='cirrus' vram='9216' heads='1'/>
</video>
</devices>
</domain>You can test the definition syntax by
virsh define /root/logger.xml
Testing the guests in KVM
Before you start your guest in KVM, I very recommend to make "dry run" test on image snapshot:
lvcreate /dev/vg_main/logger-system --snapshot -n logger-system-snap -L 2G lvcreate /dev/vg_main/logger-var --snapshot -n logger-var-snap -L 2G
(Note: /dev/vg_main/logger-* are the LVM partitions underlying the /dev/drbd5 and drbd6 devices).
Now, modify the KVM guest definition in order to use the snapshot instead of DRBD devices and start the VM by
virsh create /root/logger.xml
Switch KVM's DRBD to primary and run
If everything works OK, then modify the KVM definition back to /dev/drbd* devices and:
On XEN host, shutdown the VM:
xm shutdown logger
On XEN host, switch your DRBD devices to "secondary":
drbdadm secondary logger-system logger-var
On KVM host, switch your DRBD devices to "primary":
drbdadm primary logger-system logger-var
On KVM host, start the VM:
virsh create /root/logger.xml