Compute Clusters Integration for Debian Development and Building - Project Log
Environment Setup
Eucalyptus uses libvirt as a mechanism to interact with virtualization's hyphervisors such KVM, QEMU and others. This facility allow us to extend it's features and support arm architecture by using KVM-Qemu as emulator for libvirt's domain (Eucalyptus nc's).
In order to be able to use such facility there are few requirements that need to be met first:
The first requirement is to have kvm-qemu running as hyphervisor, so that the arm architecture is supported as a libvirt capability. Otherwise kvm will not be able to make use of qemu-system-arm as libvirt capability. For this to happen you processor should support VT (virtualization technology) for AMD-based it's called svm and for Intel-based CPUs it's called vmx. You can check more information on KVM and on VT support here.
- Second requirement is installing the required software eucalyptus software.
Virtualization facilities:
aptitude install kvm libvirt-bin qemu qemu-system
Eucalyptus packages and related software
aptitude install eucalyptus-cloud
Eucalyptus code modification
Eucalyptus branch
I've branched ?LaunchPad Eucalyptus repo, then synced with the 2.3.0 -src-offline release. Currently the Launchpad branch is outdated from such release. You can use my branch by issuing the following command:
bzr branch http://stone-head.org/debian/eucalyptus-nc-arm
To build the software you need to use pkg-eucalyptus debian/ and issue:
svn-buildpackage -rfakeroot
Related modifications
util/data.h typedef struct ncInstance_t
defines the metadata for a nc. Main fields are the following:
char instanceId[CHAR_BUFFER_SIZE]; char imageId[CHAR_BUFFER_SIZE]; char imageURL[CHAR_BUFFER_SIZE]; char kernelId[CHAR_BUFFER_SIZE]; char kernelURL[CHAR_BUFFER_SIZE]; char ramdiskId[CHAR_BUFFER_SIZE]; char ramdiskURL[CHAR_BUFFER_SIZE]; char reservationId[CHAR_BUFFER_SIZE]; char userId[CHAR_BUFFER_SIZE]; int retries;
The plan is extend it to support an arch field.
util/data.c
allocate_instance( - params - )
returns an instance.
node/handler_kvm.c Creates the instance and calls libvirt.
node/handlers.c Calls doInitialize() in node/handler_kvm.c that invokes the wrapper tool for XML domain file. It needs to make sure the arch field is being passed.
get_instance_xml invokes the wrapper tool and passes arguments to virsh.
tools/gen_kvm_libvirt_xml creates libvirt's XML Domain definition. Now it looks for an arch field to know that it needs to populate settings for ARM. However, it need the arch argument.
ARM Image
- ARM image based on current versatilepb kernel works fine and runs under qemu-system-arm. Not much trouble on building it. A nice resource is Aurel32's post.
TODO:
- Update Aurel32's howto using latest versions.
- Image cloudyfication, once the Eucalyptus support is in place.
Proposal for Image Creation Automation
Adapt VMBuilder Debian's branch to support ARM ?image creation and install so that it can later be directly uploaded to an Eucalyptus Cloud, that supports ARM images.
Approach:
Extend Debian's VMBuilder to support the versatilepb (plugins/debian/distro.py)
Debian installation for the ARM image should be automatic (preseeding).
- Install should use both kernel and initrd from Debian mirror to run the guest for installation. (qemu-system-arm ...)
- For base system install it should use either an previously downloaded ISO (preferred) or directly a netinst.
- Installation should use preseeding to completely automate it and make it unattended.
Once installed, system configuration should be done by VMBuilder's current procedure.
- Image creation finishes and it's ready for euca-upload-image.
Challenges:
VMBuilder: Clear rmadison dependency for kernel selection - https://bugs.launchpad.net/ubuntu/+source/vm-builder/+bug/560686
VMBuilder: OS installation automation. Run qemu-system-arm in "silent" mode using d-i ISO image + preseeding? *TBD*
Progress:
VMBuilder currently supports disk creation using qcow2 format (GOOD).
Command for running my updated arm image.
echo "Install Debian..." #qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.gz -hda disk.img -append "root=/dev/ram" echo "Run Debian ARM..." qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.img-2.6.32-5-versatile -hda disk.img -append "root=/dev/sda1"
Eucalyptus integration
- Eucalyptus uses libvirt as an API to interact with QEMU and KVM drivers (currently only supporting hypervisor mode) this enables to run compatible images, then creates nc (Euca nodes).
- Supported arches are x86 and x86_64.
- Supporting ARM should be feasible by hooking up with libvirt hypervisor capabilities that are already present and supported under latest Debian packages.
- Once ARM is supported image generation / automatization could be the next step if time allows.
TODO:
- DONE: Figure out how to hook up arm support using Eucalyptus libvirt handler.
- Relevant code for supporting arm on Eucalyptus is in:
- C:
- node/handlers_kvm.{c,h} - handles nc creation.
- util/data.{c,h} - handles instance creation
- Perl: tools/gen_kvm_libvirt_xml - wrapper tool for creating KVM/QEMU libvirt's XML Domain definitiion. This needs to be heavily changed/updated - redesigned, to support non-x86 archs
- C:
- There are some reported issues regarding PCI identification for qemu-system-arm, this need to be cleared later.
TODO
- Axis2 package in Debian. AXIS2_HOME should be set globally or include the jars in /usr/lib/java. Workaround: setting the env variable on .bashrc/.zshrc to avoid build errors from gatherlog/Makefile and others.
- Groovy build issues: General error during conversion: Invalid index.
Resources
http://www.aurel32.net/info/debian_arm_qemu.php - Debian ARM using Qemu