Bootstrappable Debian - Student Proposal
Name: Gustavo Prado Alkmim
Contact/Email:
alkmim on irc.debian.org #debian-bootstrap
Background:
General: In 2009, I graduated in Computer Science at the Federal University of Lavras (UFLA/Brazil) in which I worked with live-migration of virtual machines using Xen and iSCSI. I was a member of Tecnolivre, a cooperative specialized on open-source based solutions [1]. In 2012, I finished my master at State University of Campinas (Unicamp/Brazil), working with Virtual Network Mapping. Now I'm finishing the first year of my doctorate at the same university working with Mobile Cloud Computing.
Skills on Linux and programming: I started using Linux very early during my undergraduate studies. I am one of the administrators of the LRC (Network Computer Laboratory) at Computer Institute, also in Unicamp, in which we use just Debian and Ubuntu Linux systems. I have advanced skills on C++ and shell script and basic skills on Python and Java. If necessary, I have no problem starting to learn another programming language. I also had a good foundation in functional languages in my undergraduate studies using Haskell.
Experience with the proposed project: I worked with the bootstrap problem on Debian at GSoC 2011. So, I have some knowledge on how to solve cycles, how to crossbuild package and how to hack the packages in order to solve cross-build and cyclic dependencies problems. Also I learned about the staged-build process, which will be necessary to complete the project. As a result of GSoC 2011, I started hacking on Debian and the results can be found here [2]. Last year I had a good experience on patching a crosstoolchain from Ubuntu to work in Debian. The built packages may be found here [3]. This year, while preparing for GSoC I started learning Botch [9] and improved my skills on how to build packages with reduced dependencies. As a result of this study I got some patches for building packages with reduced dependencies using the Build Profiles. These patches can be found here [4].
Project title: Bootstrappable Debian
Synopsis:
The objective of this project is to enable the automated bootstrap of new Debian ports. We can summarize all the project in two items:
- Make a tool to manage bootstrap process (including instructions of how to use it) using the Botch toolchain
- Write patches and file bugs filed for all the multiarch, cross-build and profile patches needed for the bootstrap process.
As described in the project description from the Debian ideas list [5], this is a very big problem and possible all of this can't be accomplished in just one GSoC.
Benefits to Debian:
As we can see in [6] the Automatic Bootstrap is one of the going projects of Debian. As soon as this project is finished all the bootstrap process is gonna be easier even for non-experts in building Debian packages. Also, if we can do it in an automatic way, the effort to do this will be reduced. If all patches be accepted by the upstream maintainers, we will avoid solving cyclic dependencies or reduced dependencies problems every time a port is done.
Deliverables:
At the end of the Google Summer of Code, the following results should be obtained:
- An Automated Bootstrap Tool using the Botch toolchain
- Patches for all packages needed to allow the bootstrap of a base Debian system.
Project details
Small introduction
Every time a new architecture is developed, the Debian Linux needs to be ported to be used on it. Therefore, porting any operational system to a new architecture is always a difficult task since the bootstrap needs to be done 'from scratch'. While doing the first bootstrap of an architecture some challenges need to be faced. In order to do the bootstrap to a new architecture we need to have packages built that can run in the new architecture. This can be done by using multiarch or cross-built packages. Therefor we don't need to have all packages of system crossbuilt or multiarch. We just need to have a minimal build system that is sufficient to satisfy the build-dependencies of the rest of the packages.
Meanwhile, this raises another challenge that is the cyclic build-dependencies. In order to build/crossbuild packages that have cyclic build-dependencies we need to break the cycle(s) avoiding one or more build-dependencies. This is also a very difficult task because patches need to be applied. Another problem that raises when doing the bootstrap of a new architecture is while building packages inside the minimal build system. Despite all the effort in trying to crossbuild and have multiarch versions of all packages needed to solve the build-dependencies of the rest of the packages, some of them are very hard to be crossbuilt. So, we need to decide whether we should build the packages inside the minimal system or crossbuild them.
For previous ports, people tended to use non-Debian tools (such as Yocto, Gentoo and ?OpenEmbedded) to get a basic rootfs image of the target arch then do native building within that. Currently, Debian is improving its toolchain and patches for packages in order to enable the bootstrap of a Debian system using just a Debian system, instead of other OS. But a lot of work must be conclude before the bootstrap becomes easy to be done for non-expert in hacking Debian packages being able to bootstrap a Debian system. Every time someone needs to do a Debian bootstrap to a new port he needs to patch a lot of packages in order to solve all cyclic dependencies and crossbuild problems. Also, this process is manual and considering we have a high number of packages that needs to be in the basic system, the bootstrap is far from being an easy process.
Ojectives
The Bootstrappable Debian project has the objective to enable the bootstrap of new Debian ports. As explained previously it is a very difficult project since we still have a lot of problems to solve. A very good explanation about the Debian bootstrap can be found in [7] and a detailed TODO list can be found in [8].
- We can divide the project in three main works:
Build a tool that enables an automated bootstrap of new Debian ports. The main functionality of this tools is to build the packages for a Debian OS for a specified architecture from scratch. The exact parameters that the tool should manage are not defined yet, but this will be discussed during the project. One possibility is that the user should specify the set of packages that should be included in the new environment. To accomplish that, the tool should use the Botch toolchain to determine which packages are available for the minimal build system (that need to be crossbuilt or are multiarch), which packages needs to be built inside the new environment natively and the build order of those packages.
Patch all packages needed to enable the crossbuild process to be done in practice. This is a hard step of the project because to enable the crossbuild part of the process we need to modify some packages sources in Debian. This needs to be done to solve cyclic dependencies and file bugs in some packages related with the crossbuild.
Patch all packages that are going to be built inside the minimal build system. As soon as we got a minimal build system we can start building the packages of the new port natively. While building packages inside the minimal build system, we might lack some build-dependencies that have not being crossbuilt (e.g. they are very hard to crossbuild). So we need to build those packages avoiding the build-dependencies we don't have. This might be accomplished using the new Build Profile standard that enable the addition of information about staged builds.
The Build Profiles standard is the way found to insert sufficient information on debian sources packages to enable the build with the reduced build-dependencies. Through the environment variable DEB_BUILD_OPTIONS we can specify which profile to use while building a package. Build profiles extend the Build-Depends format with a syntax similar to architecture restrictions but using < and > instead. More about the Build Profiles can be found in [10].
In order to test the crossbuilt Debian system, we plain to use QEMU to emulate an armhf architecture to test if the bootstrap process for this architecture has been done successfully. After crossbuild and build all packages for the armhf architecture, we can create a repository using the built packages and use programs like debootstrap to create a new chroot to test the bootstrapped armhf architecture.
During this 2 weeks working on the project proposal to submit for the GSoC 2013, I finished some of the first patches we need to build packages inside the minimal build system with reduced build-dependencies. The patches may be found in ?[4]. Also, I started working on patches to build the Ubuntu crossbuild toolchain for the armhf architecture in Debian. These patches are not ready yet.
Project Schedule
- May 04 - Submit proposal
May 27: 19:00 UTC
- - Accepted student proposals announced on the Google Summer of Code 2013 site.
- May 27 - June 03: Prepare/Patch the crossbuild/Multiarch toolchain
- June 03 - June 10: Check which packages is known as multiarch or easily to be crossbuilt and test them. In this step I will ask for people that have already patched these packages to speedup the process.
- June 10 - June 17: Start File bugs of the packages that have failed to crossbuild.
- June 17 - July 29: File bugs in all multiarch/crossbuild packages, setup the minimal build system environment and start building/patching the native packages with reduced build-dependencies.
- July 29 - August 02: Work on Mid-Term evaluation
August 02: 19:00 UTC
- - Mid-term evaluations deadline; Google begins issuing mid-term student payments provided passing student survey is on file.
- August 02 - September 16: Project and Code the Automated Bootstrap Tool
- September 16 - September 23: Pencils down
September 23: 19:00 UTC
- - Firm 'pencils down' date. Mentors, students and organization administrators can begin submitting final evaluations to Google.
- September 23 - September 27: Work on Final Evaluation.
September 27: 19:00 UTC
- - Final evaluation deadline; Google begins issuing student and mentoring organization payments provided forms and evaluations are on file.
Other summer plans: Since I'm a phd student, I have to work on GSoC while working on my thesis, but I already have time reserved and my advisor knows about it so this will not be problem.
Exams and other commitments: I have no class or exams in the GSoC period.
Why Debian?: I choose Debian because I am familiar with the development environment. I worked in a related project in 2011 and I got very interested on the problem of make an automatic bootstrappable Debian.
* Are you applying for other projects in SoC? Yes. I'm also applying for the ?RouteFlow project but i'm giving a lot more effort to Debian project them to the other since I prefer to work on the Debian project.
* References [1] http://www.tecnolivre.com.br/ [2] http://www.lrc.ic.unicamp.br/~alkmim/gsoc2011/ [3]|http://www.lrc.ic.unicamp.br/~alkmim/debian/multiarch/ [4] http://www.lrc.ic.unicamp.br/~alkmim/debian/GSoC2013/bootstrap/patchs-alkmim/ [5] http://wiki.debian.org/SummerOfCode2013/Projects#Bootstrappable_Debian [6] http://wiki.debian.org/DebianBootstrap#Automated_Bootstrapping [7] http://wiki.debian.org/DebianBootstrap [8] http://wiki.debian.org/DebianBootstrap/TODO [9] https://gitorious.org/debian-bootstrap/botch [10] http://wiki.debian.org/DebianBootstrap#Mechanisms