Differences between revisions 207 and 208
Revision 207 as of 2012-05-28 13:45:24
Size: 18623
Editor: ?IanCampbell
Comment: Remove several references to Lenny, including bugs. Also remove some self confessed "hacky" advice in the interests of making the page easier to follow.
Revision 208 as of 2012-05-28 13:56:34
Size: 17978
Editor: ?IanCampbell
Comment: Stop giving advice to use the Xen network scripts.
Deletions are marked like this. Additions are marked like this.
Line 110: Line 110:
It is recommended that you manage your own network bridge using the [[BridgeNetworkConnections| Debian network bridge]]. The Xen supplied network scripts are not always reliable and will be removed from a later version.

Should you still wish to use them, in /etc/xen/xend-config.sxp enable the network bridge by commenting out the line that was already there for that. (You may check [[http://wiki.xensource.com/xenwiki/XenNetworking| XenNetworking]] page in Xen wiki.)

(network-script 'network-bridge antispoof=yes')

The antispoof=yes will activate Xen firewall to prevent that one of your VM uses an IP that it is not allowed to use (for example, if a domU was to use the gateway as its IP, it could seriously break your network, this will prevent it). In this case, you will need to specify the IP of your domU in the vif statement of your domUs.

If you get "missing vif-script, or network-script, in the Xen configuration file", try adding executable permission in:
chmod +x /etc/xen/scripts/*}}}

This config file also has options to set the memory and CPU usage for your dom0, which you might want to change.
It is recommended that you manage your own network bridge using the [[BridgeNetworkConnections|Debian network bridge]]. The Xen wiki page [[http://wiki.xen.org/wiki/Host Configuration/Networking|Host Configuration/Networking]] also has some useful information. The Xen supplied network scripts are not always reliable and will be removed from a later version.

The `/etc/xen/xend-config.sxp` config file has options to set the memory and CPU usage for your dom0, which you might want to change.
Line 288: Line 276:
Line 290: Line 279:
The recommended way to do this is to configure bridgine in /etc/networkin/interfaces. See [[BridgeNetworkConnections]] and/or the Xen wiki page [[http://wiki.xen.org/wiki/Host Configuration/Networking|Host Configuration/Networking]] for details. The recommended way to do this is to configure bridging in `/etc/networking/interfaces`. See [[BridgeNetworkConnections]] and/or the Xen wiki page [[http://wiki.xen.org/wiki/Host Configuration/Networking|Host Configuration/Networking]] for details.

Translation(s): Indonesian

(!) /Discussion

Xen Overview

Modern computers are sufficiently powerful to use virtualization to present the illusion of many smaller virtual machines (VMs), each running a separate operating system instance. Successful partitioning of a machine to support the concurrent execution of multiple operating systems poses several challenges. Firstly, virtual machines must be isolated from one another: it is not acceptable for the execution of one to adversely affect the performance of another. This is particularly true when virtual machines are owned by mutually untrusting users. Secondly, it is necessary to support a variety of different operating systems to accommodate the heterogeneity of popular applications. Thirdly, the performance overhead introduced by virtualization should be small.

Xen is a virtual machine monitor for x86 that supports execution of multiple guest operating systems with unprecedented levels of performance and resource isolation. Xen is Open Source software, released under the terms of the GNU General Public License. We have a fully functional ports of Linux 2.6 running over Xen, and regularly use it for running demanding applications like MySQL, Apache and PostgreSQL. Any Linux distribution (RedHat, SuSE, Debian, Mandrake) should run unmodified over the ported OS.

In addition to Linux, members of Xen's user community have contributed or are working on ports to other operating systems such as NetBSD (Christian Limpach), FreeBSD (Kip Macy) and Plan 9 (Ron Minnich).

Different types of virtualization offered by Xen

There are two different types of virtualization offered by Xen:

  • Para-virtulization and
  • Hardware-supported virtualization


A term used to describe a virtualization technique that allows the operating system to be aware that it is running on a hypervisor instead of base hardware. The operating system must be modified to accommodate the unique situation of running on a hypervisor instead of basic hardware.

Hardware Virtual Machine

A term used to describe an operating system that is running in a virtualized environment unchanged and unaware that it is not running directly on the hardware. Special hardware is required to allow this, thus the term HVM.

(Source: What is Xen Hypervisor, www.xen.org)

Domain 0 (Host) Installation

The Xen and debootstrap software in Squeeze (Debian 6.0) are very much newer than that in Lenny. Because of that, working with Xen becomes a lot easier.

The setup described here is tested for Debian Squeeze and Ubuntu Maverick virtual machines, but should work for a lot more.

First install the hypervisor, xen kernel and xen-tools. This can be done by a metapackage:

aptitude -P install xen-linux-system

To get Xen HVM support Xen 4.0 Wiki

apt-get install xen-qemu-dm-4.0

Debian Squeeze uses Grub 2 whose default is to list normal kernels first, and only then list the Xen hypervisor and its kernels.

You can change the default kernel to boot in two ways:

  • Change the priority of GRUB's Xen configuration script (20_linux_xen) to be higher than the standard Linux config (10_linux):

dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen

to undo this:

dpkg-divert --rename --remove /etc/grub.d/20_linux_xen
  • Modify the value of GRUB_DEFAULT in the file /etc/default/grub. The value is an integer starting at 0 representing the order of the menuentry item to boot. You can see a list of all menuentry values in order by typing

grep menuentry /boot/grub/grub.cfg

Count to the number of the first Xen kernel, starting at 0, and enter it in the default file. Note that this procedure is not robust against the installation or removal of new kernel versions which will cause the Xen entries to get different numbers, for this reason reordering the priorities using dpkg-divert as above is preferable.

After either of these procedures, do an update to the GRUB configuration:


If you wish to improve this process there is an open bug discussing the default values after a Xen installation. #603832

To avoid getting boot entries for each virtual machine you install on a volume group, disable the GRUB OS prober.

/!\ ToDo: does this problem still happen and under what circumstances? Bug number?

Note that if you are running a computer with multi-boot with for example Windows, this will also remove the entries for it, which might not be what you wish for.

Edit /etc/default/grub and add:

# Disable OS prober to prevent virtual machines on logical volumes from appearing in the boot menu.

You may also want to pass some boot parameters to Xen when starting up in normal or recovery mode. Add these variables to /etc/default/grub to achieve this:

# Xen boot parameters for all Xen boots
# Xen boot parameters for non-recovery Xen boots (in addition to GRUB_CMDLINE_XEN)

After editing GRUB configuration, you must apply it by running:


By default, when Xen dom0 shuts down or reboots, it tries to save the state of the domUs. Sometimes there are problems with that - it could fail because of a lack of disk space in /var, or because of random software bugs. Because it is also clean to just have the VMs shutdown upon host shutdown, if you want you can make sure they get shut down normally by setting these parameters in /etc/default/xendomains:


It is recommended that you manage your own network bridge using the Debian network bridge. The Xen wiki page Host Configuration/Networking also has some useful information. The Xen supplied network scripts are not always reliable and will be removed from a later version.

The /etc/xen/xend-config.sxp config file has options to set the memory and CPU usage for your dom0, which you might want to change. To reduce dom0 memory usage as it boots, use the dom0_mem kernel option in the aforementioned GRUB_CMDLINE_XEN variable. Xen wiki also advise to disable dom0 memory ballooning and set minimal memory in /etc/xen/xend-config.sxp (1024M is an example) :

(dom0-min-mem 1024)
(enable-dom0-ballooning no)

Serial console access

To get output from GRUB, the Xen hypervisor, the kernel and getty (login prompt) via both VGA and serial console to work, here's an example of the right settings on squeeze:

Edit /etc/default/grub and add:

GRUB_SERIAL_COMMAND="serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1"
GRUB_TERMINAL="console serial"
GRUB_CMDLINE_XEN="com1=9600,8n1 console=com1,vga"
GRUB_CMDLINE_LINUX="console=tty0 console=hvc0"

Here's what I used to configure the serial console (for a Supermicro X8STi-F motherboard with IPMI and SOL):

GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all com1=115200,8n1,0x3e8,5 console=com1,vga"
GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen"

In /etc/inittab you need at least these lines:

1:2345:respawn:/sbin/getty 38400 hvc0
2:23:respawn:/sbin/getty 38400 tty1
# NO getty on ttyS0!

This way, tty1 will show up at the VGA output, and the hvc0 will show up at the serial console.

To keep both Xen and dom0 kernel output on the same tty, just omit the "vga"-related settings from the above setup.

If you need to debug Xen and see a crash dump of the kernel, you can do it using IPMITool if your server has SOL:

ipmitool -I lanplus -H server-ip-address -U your-username sol activate | tee my-log-file.txt

Installation as a DomU (guest)

Using xen-tools

xen-tools is a set of scripts which can easily create fully configured Xen guest domains.

Once you have installed dom0 you can install xen-tools on your host with:

apt-get install xen-tools

Then you can create virtual machines with this command:

xen-create-image --hostname <hostname> --ip <ip> --vcpus 2 --pygrub --dist <lenny|squeeze|maverick|whatever>

To configure xen-tools, you can edit /etc/xen-tools/xen-tools.conf which contains default values that the xen-create-image script will use. The xen-create-image(8) manual page contains information on the available options.

Possible problems and bugs

  • If your domU kernel happens to miss support for the xvda* disk devices (the xen-blkfront driver), use the --scsi option that makes the VM use normal SCSI HD names like sda*. You can also set scsi=1 in /etc/xen-tools/xen-tools.conf to make this the default.

  • When using xen-tools with --role on Squeeze be aware of #588783: 'Should mount /dev/pts when creating image'. This is fixed in Wheezy.

Using Debian Installer

The Xen wiki page Debian Guest Instalation Using DebianInstaller contains instructions on how to install Xen DomU from Lenny onwards using ?Debian Installer.


See also: Debian Release Notes

Upgrading a server to Squeeze that uses both Lenny Dom0 and DomU's is fairly straightforward. There are a few catches that one needs to be aware of however: Reference

  • Dom0 Issues

    • The Xen packages will not upgrade themselves. They must be manually removed and the latest Xen packages must be installed from the Debian Squeeze repository through apt.
    • pygrub in Xen-4.0 will need to be patched as per #599243

    • xen.independent_wallclock sysctl setting is not available for newer squeeze kernels supporting pvops. If you have relied on it, you would have to run ntpd in dom0 and domUs. source

  • DomU Issues

    • A Squeeze DomU will not be able to boot on the Xen-3.2 package supplied by Lenny because this older version will not support grub2. A Lenny DomU can be upgraded to Squeeze while running on a Lenny Dom0 but it will not be able to be booted until the Dom0 has been upgraded to the Xen-4.0 packages.
    • The entries added to chain load grub1 to grub2 will not allow pygrub to find the correct partition. Before rebooting a freshly upgraded Squeeze DomU, make sure to rename or remove /boot/grub/menu.lst. This will force pygrub to look for the /boot/grub/grub.cfg file which will be in the correct format.
    • A qcow image created with qcow-create and the BACKING_FILENAME option on Lenny will not be able to boot on Squeeze because the ability to use qcow images as backing files has been removed in Xen versions after 3.2. Also, if you try to boot such an image on Squeeze, Xen will silently convert the qcow images L1 table to big endian (you'll find "Converting image to big endian L1 table" in the logfiles), effectively rendering the image unusable on both Squeeze and Lenny!

Note on kernel version compatibility

The new 2.6.32 kernel images have paravirt_ops-based Xen dom0 and domU support.

When you create an image for a modern Debian or Ubuntu domU machine, it will include a kernel that has pv_ops domU support, it will therefore not use a Xen kernel, but the "stock" one, as it is capable of running on Xen's hypervisor.

Possible problems and bugs

'clocksource/0: Time went backwards'

If a domU crashes or freezes while uttering the famous lasts words 'clocksource/0: Time went backwards', your domU is likely using the xen clocksource instead of its own clock ticks. In practice, this seems to be the cause of infrequent lockups under load (and/or problems with suspending).

see http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1098

workaround #1

A workaround is to decouple the clock in the domU from the dom0:

In your dom0 and domU /etc/sysctl.conf add the line: xen.independent_wallclock=1. On the dom0, edit the configuration file of the domU (e.g. /etc/xen/foobar.cfg and add (or expand) the extra-line: extra="clocksource=jiffies".

These settings can be activated without rebooting the domU. After editing the configuration files, issue sysctl -p and echo "jiffies"> /sys/devices/system/clocksource/clocksource0/current_clocksource on the domU prompt.

Because the clock won't be relying on the dom0 clock anymore, you probably need to use ntp on the domU to synchronize it properly to the world.

workaround #2

Another possibility ist to use the behaviour of the previous xen-kernel settings: clocksource=jiffies and independent_wallclock=0

Setting clocksource=jiffies for the dom0 and each domU as kernel parameter has eliminated the "Time went backwards" for me (14 dom0s and 27 domUs running stable for two weeks). You can check the values with

cat /sys/devices/system/clocksource/clocksource0/current_clocksource


cat /proc/sys/xen/independent_wallclock

With these settings, ntp ist only needed in the dom0. If you change the time in a domU while ntp is running on the according dom0, time will be corrected within a few minutes in the domU. Hint: I didn't manage to influence the time of the domU with setting the time in the dom0 with date or hwclock, nevertheless ntp seems to do this (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=534978#29).

workaround #3

There are cases where setting the clocksource to jiffies just makes the clock more unstable and leads to continous resets. A working solution appears to be the following:

  • set independent_wallclock to 0 (all domains; VMs will follow dom0's clock)
  • set clocksource to xen (it's the default in lenny)
  • configure ntpd in dom0 only; set "disable kernel" in ntp.conf

This succeeded in stabilizing a Xen server's clock where all other workarounds failed.

More information can be found at http://my.opera.com/marcomarongiu/blog/2010/08/18/debugging-ntp-again-part-4-and-last. You can browse for the whole process at http://my.opera.com/marcomarongiu/blog/index.dml/tag/Sysadmin

Older Releases

Xen Installation on Debian 5.0 ( Lenny )

Xen Installation on Debian 4.0 ( Etch )

The page DebianInstaller/Xen contains instructions on how to install Xen Dom0 and Etch DomU with DebianInstaller.

Package maintenance

Debian's Xen packages are maintained by the pkg-xen project. (developers' mailing list)

The Debian Developer's Package Overview page lists source packages that are maintained by the team.

Common Errors

dom0 automatic reboots

  • {i} Note: if Xen is crashing and reboot automatically, you may want to use noreboot xen option, to prevent it from rebooting automatically.

Edit /etc/default/grub and add the "noreboot" option to GRUB_CMDLINE_XEN, for example:


Error "Device ... (vif) could not be connected"

You need to configure some basic networking between dom0 and domU.

The recommended way to do this is to configure bridging in /etc/networking/interfaces. See BridgeNetworkConnections and/or the Xen wiki page Host Configuration/Networking for details.

  • {i} Note: The use of the 'network-script' option in /etc/xen/xend-config.sxp is no longer recommended.

PV drivers on HVM guest

It may be possible to build the PV drivers for use on HVM guests. These drivers are called unmodified_drivers and are part of the xen-unstable.hg repository. You can fetch the repository using mercurial thus:

  •   hg clone http://xenbits.xen.org/xen-unstable.hg

The drivers reside under xen-unstable.hg/unmodified_drivers/linux-2.6. The README in this directory gives compilation instructions.

A somewhat dated, detailed set of instructions for building these drivers can be found here: