Preparation of an Eucalyptus-ready cloud image

This page describes very down to earth how to create a Debian image, -kernel and -initrd triplet for Eucalyptus cloud computing on the basis of KVM. It uses only commonly known tools, nothing fancy, and is thus practical and educational. This description was performed by Dominique (<domibel>) as part (or a bit beyond) of his 2010 Google Summer of Code project with some ornamental editing by Steffen (<moeller>). This description effectively achieves the same status that vmbuilder would achieve. However, we find it easier and with the skills laid out one can expect to apply this in various other situations.

To present an overview on the process:

The boxes present code that should be executed on your machine. Since we have two machines, this may be irritating at a first read. To allow an easy copy'n'paste, only the short note "physical" or "virtual" is presented on top of every box. Caveat: you never know if this page was possibly modified by someone with a somewhat strange kind of humor. Since quite a few commands are executed as root, truly try to understand what every line is doing.

Retrieve packages for building, other preparations

The following packages should be available to follow these instructions. Also check that you have sufficient disk space available. You need twice the amount of free space available that you want to reserve ask disk space for your image, i.e. at least 2*2GB.

physical:

df

The following yo need for the installation

physical:

apt-get install qemu-kvm xtightvncviewer wget

Create your own image

Debian offers images with compilations of its latest packages, readily available as burnable ISO files. Further down we will start those on a virtual image to perform a regular Debian installation - on a virtual image. Here we are proposing to download the very latest snapshot of the testing distribution (which is closest to our heart as Debian developers), now so close to the release of the next version those should work (tested June 24th, 2010 and again August 4th), consider falling back to an official release if you experience difficulties. The following blocks should yield a readily upload-able virtual image, if you only paste the command lines to a shell. You need root privileges at some point.

Please decide for a Debian architecture to use and give a name to the image.

physical:

#arch="i386"
arch="amd64"
rawimg="raw.img"
rootimg="root.img"

Now perform the download of the Debian installer ISO.

physical:

wget http://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/${arch}/iso-cd/debian-testing-${arch}-netinst.iso

Create the raw disk. Note, that we experienced difficulties with the qcow2 format, hence the "-f raw".

physical:

[ -n "$rawimg" ] || rawimg="raw.img"
kvm-img create -f raw $rawimg 2G

Running the installer

Now boot the netboot CD image. It will greet you (as if you booted from a regular machine) with the Debian installer - but - you don't see it, because we (and this is what we want) gave it the "-nographic" option. Why? Because everyone is doing it, feel free to edit this wiki page and give a proper reason that goes beyond a "we could execute this remotely without installing X".

physical:

[ -n "$rawimg" ] || rawimg="raw.img"
sudo kvm -m 256 -cdrom debian-testing-${arch}-netinst.iso -drive file=$rawimg,if=scsi,index=0 -boot d -net nic -net user -nographic -vnc :0

physical: ***experimental with virtio support***

[ -n "$rawimg" ] || rawimg="raw.img"
sudo kvm -m 256 -cdrom debian-testing-${arch}-netinst.iso -drive file=$rawimg,if=virtio -boot d -net nic,model=virtio -net user -nographic -vnc :0

Keep the above process running, start the platform-independent viewer in another shell. If running on the same host (expected) you can leave the field for the hostname to connect empty.

physical:

xtightvncviewer

The Debian installer has become very user friendly. When preparing a cloud image one has some ideas on things to change, well, it is not too bad.

Starting and Cloudification of the created image

After the installation the thus prepared disk can be booted:

physical:

[ -n "$rawimg" ] || rawimg="raw.img"
sudo kvm -m 256 -drive file=$rawimg,if=scsi,index=0,boot=on -boot c -net nic -net user -nographic -vnc :0 -redir tcp:5555:10.0.2.15:22

physical: ***experimental with virtio support***

[ -n "$rawimg" ] || rawimg="raw.img"
sudo kvm -m 256 -drive file=$rawimg,if=virtio,boot=on -boot c -net nic,model=virtio -net user -nographic -vnc :0 -redir tcp:5555:10.0.2.15:22

This previous vnc connection will have terminated. Start it again to see the Debian login prompt or investigate problems of various sorts. To have copy and paste, however, connect via ssh:

physical:

 ssh -X -p 5555 guest@localhost

We need to do some cleanup first, we suggest to remove the link to the cdrom, even though it is not ultimately required. Execute this (like the following commands) from a root shell of the virtual image:

virtual:

 vi /etc/apt/sources.list

The following packages shall form the basic stock for the virtual image.

virtual:

 aptitude install curl devscripts nmap locales ntpdate cvs subversion git cmake-curses-gui

Of the above, we somehow need

For MPI programming also install

virtual:

 aptitude install openmpi-dev openmpi-bin

For the queueing system, which this project is about, we will have a separate section. The packages for the queueing system shall not be part of the bare image but be installed after the image is already running, i.e. with all information on IP numbers and hostnames available.

replace /etc.rc.local

virtual:

cat <<EORCLOCAL > /etc/rc.local
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# load pci hotplug for dynamic disk attach in KVM (for EBS)
depmod -a
modprobe acpiphp

# simple attempt to get the user ssh key using the meta-data service
mkdir -p /root/.ssh
echo >> /root/.ssh/authorized_keys
curl -m 10 -s http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key | grep 'ssh-rsa' >> /root/.ssh/authorized_keys
echo "AUTHORIZED_KEYS:"
echo "************************"
cat /root/.ssh/authorized_keys
echo "************************"
exit 0
EORCLOCAL

The following is not completely tested

For MPI, on user account: guest

 ssh-keygen -t rsa

Test, not sure if this works

 cat /home/guest/.ssh/id_rsa.pub >> .ssh/authorized_keys
 chmod 700 .ssh/authorized_keys
 chmod 700 .ssh

Preparation of the kernel and initrd images

Our current virtual image is apparently functional. And knowing about the inter-dependencies of kernels and modules, we don't want to fiddle with that but just use exactly those files, please. Adjustments can still be performed within that root image with the Debian-typical means, which we don't want to explain here.

Copy initrd and vmlinuz to your front end:

virtual:

# say what username to copy to
[ -n "$yourid" ] || yourid=root
# determine the IP address of your physical node, which here acts as the gateway
youripnum=$(/sbin/route -n | awk '{print $2}' | egrep -v '^(IP|Gateway|0\.0\.0\.0)')
# but just chose any IP address of a machine that knows how to upload to Eucalyptus
scp /boot/initrd.img-$(uname -r) /boot/vmlinuz-$(uname -r) ${yourid}@${youripnum}:

Delete the line "127.0.1.1 debian" in /etc/hosts for now, we will add one later again when we know them.

virtual:

vi /etc/hosts

For squeeze you have to delete the following file. It is supposed to somehow memorize your network interface, but this has felt unreliable.

virtual:

rm -rf /etc/udev/rules.d/70-persistent-net.rules

Don't reboot just now! The file will be regenerated after reboot by /lib/udev/write_net_rules . Instead, shutdown

virtual:

shutdown -h now

and extract your root file system's image as described below:

physical:

sudo parted $rawimg

When in parted, perform

physical:

unit b
print

You will see something analogous to the following table:

Number  Start        End          Size         Type      File system     Flags
 1      1048576B     1988100095B  1987051520B  primary   ext3            boot
 2      1989147648B  2146435071B  157287424B   extended
 5      1989148672B  2146435071B  157286400B   logical   linux-swap(v1)

Leave with

physical:

quit

The first partition, does not start at block 0 or block 1, but there is some offset, indicated in bytes. With 512 bytes per block, we know the number of blocks to be skipped to reach the first _real_ byte.

Number of blocks to skip:

1048576 / 512 = 2048

Number of blocks to read (end-start)/size

(1988100095 - 1048576) / 512 = 3880960

Now do the actual transfer of the contents, please adapt the size of your disk.

physical:

[ -n "$rootimg" ] || rootimg="root.img"
[ -n "$rawimg" ] || rawimg="raw.img"
dd if=$rawimg of=$rootimg bs=512 skip=2048 count=3880960

for our 2G image this looks like (to be executed on a bash shell)

physical:

[ -n "$rootimg" ] || rootimg="root.img"
[ -n "$rawimg" ] || rawimg="raw.img"
dd if=$rawimg of=$rootimg bs=512 skip=$((1048576/512)) count=$(( (1988100095-1048576)/512 ))

Now this triplet (rootfs, vmlinuz, initrd) shall all be tested in combination. If they are still in that folder owned by the user previously indicated by '$yourid', then move them to your local directory for the sake of simplicity:

physical:

[ -n "$yourid" ] || yourid=root
[ -n "$arch" ] || arch=amd64
mv  ~$yourid/initrd.img-2.6.32-5-$arch ~$yourid/vmlinuz-2.6.32-5-$arch .

It is those triplets that we will then also explicitly specify for the Eucalyptus images. Since we are using the extracted rootFS, our rootFS has changed from /dev/sda1 to /dev/sda. Please adjust for your respective kernel version, which may have increased when you read this / you may have a different platform:

physical:

[ -n "$rootimg" ] || rootimg="root.img"
[ -n "$arch" ] || arch=amd64
sudo kvm -m 256 -hda $rootimg -net nic -net user -nographic -vnc :0 \
         -initrd initrd.img-2.6.32-5-$arch \
         -kernel vmlinuz-2.6.32-5-$arch \
         -append "rootwait root=/dev/sda"

Please adjust the kernel version and architecture for your respective local setup. Now, with the restart, that unfortunate persistency helper file was created again. Please remove it again. Now.

virtual:

rm -rf /etc/udev/rules.d/70-persistent-net.rules

and also (just in case you did something sensitive) also remove your bash history, which is created again when booting next.

virtual:

rm ~/.bash_history

For some further clean, i.e. everything we clean will safe us money or time with every instance created, remove whatever you want to remove.

virtual:

aptitude clean
halt

Some tricks

The root.img file is a regular file. This has several positive side-effects.

Preparation of variants of the image

You want to prepare a particular bioinformatics or astronomy or ... image with the here described variant as a basis? Just

physical:

[ -n "$rootimg" ] || rootimg="root.img"
orig=$rootimg
new="medbio.img"
cp $orig $new

and for the parallel management of the various such images with the here presented command line excerpts:

physical:

rootimg="$new"

Loop-mount to local file system

The image that can be mounted to the local machine.

physical:

[ -n "$rootimg" ] || rootimg="root.img"
mkdir tmp_mnt
sudo mount -o loop $rootimg tmp_mnt && ls tmp_mnt

If successful, one will see the root directory of root.img under tmp_mnt. That mounting can be used to change files, too, e.g. for

physical:

rm tmp_mnt/etc/udev/rules.d/70-persistent-net.rules

to unmount perform

physical:

umount tmp_mnt

Some more bits are possible, e.g. one can run

physical:

sudo chroot tmp_mnt

and even install packages without virtualization But this is unsafe, rather use the virtual image.

Final steps for an image on the Cloud Server

Bundling and uploading to Eucalyptus

We can now proceed as it is described in a regular cloud computing text book, i.e. in the documentation to Eucaluptus. From your physical machine, with the euca2ools package installed, perform the following

physical:

# if the images copied from your virtual machine are not yet in the CWD
# then add links to those or copy them to here.
euca-bundle-image -i vmlinuz-2.6.32-5-$arch --kernel true
euca-upload-bundle -b $arch -m /tmp/vmlinuz-2.6.32-5-$arch.manifest.xml
euca-register $arch/vmlinuz-2.6.32-5-$arch.manifest.xml
EKI=eki-04B810D0

euca-bundle-image -i  initrd.img-2.6.32-5-$arch
euca-upload-bundle -b $arch -m /tmp/initrd.img-2.6.32-5-$arch.manifest.xml
euca-register $arch/initrd.img-2.6.32-5-$arch.manifest.xml
ERI=eri-3BB111B0

We set the name of the bundle to the name of the architecture. But any name could be used.

Now, the prior two images are both combined with the root image. Above EKI/ERI numbers are ours, yours will differ. If you are continuing from another shell or for another image, those EKI/ERI IDs can be rerieved any time from 'euca-describe-images'.

physical:

[ -n "$rootimg" ] || rootimg="root.img"
[ -n "$arch" ] || arch="amd64"
euca-bundle-image -i $rootimg --kernel $EKI --ramdisk $ERI
euca-upload-bundle -b $arch -m /tmp/${rootimg}.manifest.xml
euca-register $arch/${rootimg}.manifest.xml
EMI=emi-1B1B0C94

The images are now inspect- and available from your Eucalyptus server. One can also see, that the root/machine image (EMI) is connected to the kernel (EKI) and initrd/ramdisk (ERI) images.

physical:

$ euca-describe-images
IMAGE   eki-D224100C    amd64/vmlinuz-2.6.32-5-amd64.manifest.xml       guest available       public          x86_64  kernel
IMAGE   emi-1B1B0C94    amd64/root.img.manifest.xml     guest available       public          x86_64  machine eki-D224100C    eri-059910F2
IMAGE   eri-059910F2    amd64/initrd.img-2.6.32-5-amd64.manifest.xml    guest available       public          x86_64  ramdisk

To start, we just start, as we did with the offcial Eucalyptus images before

physical:

 $ euca-run-instances -k mykey $EMI
RESERVATION     r-35140741      guest guest-default
INSTANCE        i-28CC058F      emi-1B1B0C94    0.0.0.0 0.0.0.0 pending mykey   0               m1.small        2010-08-04T16:48:45.983Z        debian-rocks     eki-D224100C    eri-059910F2
$ euca-describe-instances i-28CC058F
RESERVATION     r-35140741      guest default
INSTANCE        i-28CC058F      emi-1B1B0C94    0.0.0.0 0.0.0.0 pending mykey   0               m1.small        2010-08-04T16:48:45.983Z        debian-rocks     eki-D224100C    eri-059910F2

and after some time, with the dhcpd been contacted

physical:

$ euca-describe-instances i-28CC058F
RESERVATION     r-35140741      moeller default
INSTANCE        i-28CC058F      emi-1B1B0C94    192.168.100.102  192.168.100.102  running mykey   0               m1.small        2010-08-04T16:48:45.983Z        debian-rocks     eki-D224100C        eri-059910F2

Considerations for a ''production'' image

Caveat: The way the instance is currently set up, once the machine runs in Eucalyptus, one can always log in with the root password and the guest account that was set during installation. For an instance that is shared among a group of individuals, this is not appropriate:

See also

References