Translation(s): none


Migration of VMs between Xen and KVM - back and forth - the simple way

I am writing this in English because I spent one whole day by googling and I found no info about this even being possible. So I spent another day by frustrating experiments with no results, only to find very simple way which made me happy.

What we want to achieve

Migration of VMs between two Xen and KVM machines.

The migration does not have to be online, but it must be quick and easy.

The migration is possible because XEN is great and flexible system, allowing to customize all VMs in a way that they can be started in both XEN and KVM with no intermediate steps, so after the "one-time" modifications described below, no other VM conversions or modifications are necessary for the migrations.

The Xen is Debian 5 Lenny, KVM is Ubuntu 10.04 (~Debian Squeeze).

The XEN VM guests were created by the xen-create-image tool and we have too many of them, some with Debian 4, some with Debian 5, some with Debian 6. If all of these should be reinstalled from scratch, that would take month of work!

I do not guarantee that this will work in XEN 4 (Debian Squeeze XEN host), because I can't try it currently.

Why I need this

Because:

How to do it

First of all, setup and start the DRBD devices, which will replicate your VM disks from the XEN to the KVM physical host machines.

Then, you must customize the XEN VM guests the following way:

/etc/inittab in all VM guest

You must comment out the following line, which is created by xen-create-image and directs console to the hvc0, which does not work in KVM:

 #1:2345:respawn:/sbin/getty 38400 hvc0

Instead, place the standard tty1 console definition there:

 1:2345:respawn:/sbin/getty 38400 tty1

Now, the console will not work in XEN. For enabling XEN console, you can place another line:

 co:23:respawn:/sbin/getty -L hvc0 9600 vt102 

Or you can redirect the hvc0 console to the tty1 by placing additional line in the VM guest definition file (e.g. /etc/xen/auto/logger.cfg)

 extra = "console=hvc0 xencons=tty"

Modify VM guest definition files

You must rename your disks to sda/sdb/sdc... in the guest definition files (/etc/xen/auto/*.cfg).

By default, xen-create-image generates disk names with partition numbers in the form (hd|sd|xvd)[a-z][0-9], e.g. hda1, sda1 or xvda1

The problem is, that

Therefore the only possible way forward is to use the sda/sdb/sdc... disk names in XEN.

Fortunatelly, XEN is happy with sda/sdb/sdc disk names so you can modify the guest definition files to like this:

 #
 # Configuration file for the Xen instance logger, created
 # by xen-tools 3.9 on Wed May 19 02:56:03 2010.
 #
 #
 #  Kernel + memory size
 #
 kernel      = '/boot/vmlinuz-2.6.26-2-xen-amd64'
 ramdisk     = '/boot/initrd.img-2.6.26-2-xen-amd64'
 memory      = '2048'
 
 #
 #  Disk device(s).
 #
 root        = '/dev/sda ro'
 disk        = [
                   'phy:/dev/vg_main/logger-swap,sdc,w',
                   'phy:/dev/drbd5,sda,w',
                   'phy:/dev/drbd6,sdb,w',
               ]
 cpus="0-7"
 vcpus=8
 extra="clocksource=jiffies"
 #for standard /etc/inittab (compatible with KVM):
 extra = "console=hvc0 xencons=tty"
 #
 #  Hostname
 #
 name        = 'logger'
 #
 #  Networking
 #
 dhcp        = 'dhcp'
 vif         = [ 'mac=00:16:3E:37:D0:70' ]
 
 #
 #  Behaviour
 #
 on_poweroff = 'destroy'
 on_reboot   = 'restart'
 on_crash    = 'restart'

Change /etc/fstab in each guest VM

Before you restart (re-create) the guest VM, log into it and modify its /etc/fstab mount points to comply with your new disk names !

Install grub and kernel in each VM guest

Install Grub - it should be happy now with your /dev/sda root disk:

 apt-get install grub2

Install the proper kernel file - e.g., for Debian Squeeze VM guest:

 apt-get install linux-image-2.6.32-5-amd64

Make sure that the /boot/grub/device.map contains this line:

 (hd0)   /dev/sda

and then run

 update-grub

which should output something like

Updating /boot/grub/grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.26-2-amd64
Found initrd image: /boot/initrd.img-2.6.26-2-amd64
done

Re-create and test the guest VM in XEN

Either

 poweroff

in guest or

 xm shutdown logger

in host, and then

 xm create -c /etc/xen/auto/logger.cfg

Create new guest defintion file for KVM

On the KVM machine, thanks to DRBD, you already have the guest partitions ready, so the last step is to create the proper guest definitions, e.g. /root/logger.xml :

 <domain type='kvm'>
  <name>logger</name>
  <uuid>30ea508e-7ed4-d590-3f63-0ea9d22e2545</uuid>
  <memory>2097152</memory>
  <currentMemory>2097152</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.12'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/drbd5'/>
      <target dev='sda' bus='ide'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/drbd6'/>
      <target dev='sdb' bus='ide'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vg_main/logger-swap'/>
      <target dev='sdc' bus='ide'/>
    </disk>
    <interface type='bridge'>
        <mac address='00:16:3E:37:D0:70'/>
      <source bridge='br0'/>
      <model type='virtio'/>
    </interface>
    <console type='pty'>
      <target port='0'/>
    </console>
    <console type='pty'>
      <target port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
  </devices>
 </domain>

You can test the definition syntax by

 virsh define /root/logger.xml

Testing the guests in KVM

Before you start your guest in KVM, I very recommend to make "dry run" test on image snapshot:

 lvcreate /dev/vg_main/logger-system --snapshot -n logger-system-snap -L 2G
 lvcreate /dev/vg_main/logger-var --snapshot -n logger-var-snap -L 2G

(Note: /dev/vg_main/logger-* are the LVM partitions underlying the /dev/drbd5 and drbd6 devices).

Now, modify the KVM guest definition in order to use the snapshot instead of DRBD devices and start the VM by

 virsh create /root/logger.xml

Switch KVM's DRBD to primary and run

If everything works OK, then modify the KVM definition back to /dev/drbd* devices and:

On XEN host, shutdown the VM:

 xm shutdown logger

On XEN host, switch your DRBD devices to "secondary":

 drbdadm secondary logger-system logger-var

On KVM host, switch your DRBD devices to "primary":

 drbdadm primary logger-system logger-var

On KVM host, start the VM:

 virsh create /root/logger.xml