Differences between revisions 3 and 9 (spanning 6 versions)
Revision 3 as of 2010-04-06 20:08:41
Size: 9714
Editor: ?JeremieKoenig
Comment: improvements after youpi's comments, summarize the bootloader situation
Revision 9 as of 2010-05-26 10:22:09
Size: 12544
Editor: ?JeremieKoenig
Comment: add real dates
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
'''Mentor:''' Samuel Thibault (youpi) <sthibault@debian.org>
Line 6: Line 8:

'''Update 2010-05-24''': The proposal has been accepted by Debian. I will keep track of my progress on the [[SummerOfCode2010/HurdDebianInstaller/JeremieKoenig/Roadmap]] page.
Line 15: Line 19:
As for hurd I am less familiar with its guts, but I am enthusiastic about its overall design and I had tried it in the past. Preparing this application also involved some looking under the hood and tinkering with it, so I am confident that I would be able to acquire the necessary knowledge without too much of a hassle. As for the Hurd I am less familiar with its guts, but I am enthusiastic about its overall design and I had tried it in the past. Preparing this application also involved some looking under the hood and tinkering with it, so I am confident that I would be able to acquire the necessary knowledge without too much of a hassle.
Line 18: Line 22:

Hurd is micro-kernel based system:
most of the code usually found in the kernel of Unix-like systems
is implemented in userspace,
as a series of cooperating server processes.
The filesystem is used as a namespace for the services that the system provides
(the servers are said to ''translate'' parts of the filesystem),
and the principle of ''capabilities'' is used to manage access to resources.
This design offers great flexibility,
as even a normal user can use to own servers,
in addition to or as a replacement to the system ones.
Line 29: Line 44:
Deliverables
Line 31: Line 45:
My work would primarily consist of changes to existing software. I would submit them as I go, as patches against Debian packages and upstream where appropriate. I would also keep track of them to submit to Google after GSoC is over. == Deliverables ==

My work would primarily consist of changes to existing software.
I would submit them as I go, as patches against Debian packages and upstream where appropriate.
I would also keep track of them, so I can submit them
to Google after GSoC is over.
Line 37: Line 55:
Samuel Thibault has some [[http://people.debian.org/~sthibault/hurd-i386/|packages and patches]] which can be used to build "monolithic" debian-installer images, which at some point he could boot into a shell. I have fixed a [[http://bugs.debian.org/576519|bug in ext2fs]] which prevents them from booting but as of 2010-04-06, another problem with the bootstrap process remains (namely, it hangs after the exec server is started by ext2fs.) Samuel Thibault has some [[http://people.debian.org/~sthibault/hurd-i386/|packages and patches]] which can be used to build "monolithic" debian-installer images, which at some point he could boot into a shell. I have fixed a [[http://bugs.debian.org/576519|bug in ext2fs]] which prevented them from booting but as of 2010-04-06, another problem with the bootstrap process remains (namely, it hangs after the exec server is started by ext2fs.)
Line 41: Line 59:
I have [[http://lists.debian.org/debian-hurd/2010/03/msg00049.html|started working]] on "clean" patches for porting busybox. My first patches have been accepted upstream with some changes and the source from the upstream git repository builds on Hurd with a minimal config. However more applets will probably need to be ported as I progress with the rest of the installer. I have [[http://lists.debian.org/debian-hurd/2010/03/msg00049.html|started working]] on "clean" patches for porting busybox. My first patches have been accepted upstream with some changes and the source from the upstream git repository builds on Hurd with a minimal configuration.
However more applets will probably be needed as I progress with the installer.
Line 43: Line 62:
In any case, some applets will need to be disabled in the Debian package on Hurd and kFreeBSD. This could be done either by using system-specific configuration files in the Debian package, or by disabling the relevant applets in the build system using KConfig dependencies. In any case, some applets will need to be disabled in the Debian package on Hurd and kFreeBSD.
This could be done either by using system-specific configuration files in the Debian package,
or by modifying the build system of busybox:
the current target system would be detected,
and the irrelevant applets would be disabled through KConfig dependencies.

=== Genext2fs ===

The ext2fs translator only supports filesystems with 4KB blocks right now. On the other hand, genext2fs [[http://bugs.debian.org/526999|uses a fixed block size]] of 1024. The problem could be attacked on either side, but the genext2fs route seems much more simple. Samuel Thibault has [[http://people.debian.org/~sthibault/hurd-i386/genext2fs.patch|a patch]] which changes the hardwired #define from 1024 to 4096, and sets the superblock's "creator OS" field to a system-specific value. In the longer term, however, these would need to be turned into command-line options.
Line 51: Line 78:
=== Genext2fs ===

The ext2fs translator only support filesystems with 4KB blocks right now. On the other hand, genext2fs [[http://bugs.debian.org/526999|uses a fixed block size]] of 1024. The problem could be attacked on either side, but the genext2fs route seems much more simple. Samuel Thibault has [[http://people.debian.org/~sthibault/hurd-i386/genext2fs.patch|a patch]] which changes the hardwired #define from 1024 to 4096, and sets the superblock's "creator OS" field to a system-specific value. In the longer term, however, these would need to be turned into a command-line options.
Line 59: Line 82:
I suspect udeb tuning and further changes might prove necessary as well. (for instance there are currently workarounds for the installer in the Debian package's libstore bootstrap code and I strongly suspect that they will need some changes or cleaning up). Tuning hurd-udeb and further changes might be necessary as well.
For instance, there are currently workarounds for the installer in the Debian package's libstore bootstrap code and I strongly suspect that they will need some changes or cleaning up.
Line 63: Line 87:
Apparently, netcfg has had Hurd support for some time (r548, by David Whedon starts it, the more recent r62649 by Samuel Thibault adds some fixes). However, I'm not sure that DHCP is supported, which it also isn't in a straightforward way on an installed Hurd system, as I understand it. Apparently, netcfg has had Hurd support for some time (r548, by David Whedon starts it, the more recent r62649 by Samuel Thibault adds some fixes). However, I'm not sure that DHCP is supported, which it also isn't in a straightforward way on an installed Hurd system.
Line 66: Line 90:
For the record, Samuel Thibault has submitted [[http://bugs.debian.org/555252|a patch]]
which adds Hurd support to dhcp3.
Line 70: Line 96:
As a consequence, we have to use ext2 as the root filesystem, with special options at mkfs time as mentioned above for genext2fs.
More generally, Hurd has only has a subset of the filesystems which Linux understands.
As a consequence, we have to use ext2 as the root filesystem,
with special options at mkfs time as mentioned above for genext2fs.
More generally, Hurd only has a subset of the filesystems which Linux understands.
Line 84: Line 111:
The [[http://www.gnu.org/software/hurd/hurd/console.html|Hurd console]] has some UTF-8 support and its VGA driver can use any 512 glyphs at once in a dynamic fashion. So while I don't expect it to handle bidi or large glyphs, I believe it could be used for a fair number of languages nontheless. Mach has a very basic console which could be used in the beginning.
Line 86: Line 113:
On the other hand, the VGA driver uses BDF fonts, so a udeb similar to bterm-unifont would have to be created. Also, the keyboard driver does not support any kind of keymaps so this would need to be implemented, and integrated with kdb-chooser and read from /etc/default/keyboard-configuration. Among other interesting features,
the [[http://www.gnu.org/software/hurd/hurd/console.html|Hurd console]] has some UTF-8 support
and its
VGA driver can use any 512 glyphs at once in a dynamic fashion.
So while I don't expect it to handle bidi or large glyphs,
I believe it could be used for a fair number of languages nontheless.

The Hurd console
uses BDF fonts, so a udeb similar to bterm-unifont would have to be created.
Also, the keyboard driver does not support any kind of keymaps so this would need to be implemented,
and integrated with kdb-chooser and /etc/default/keyboard-configuration.
Line 90: Line 125:
Right now, only the "netboot" and "monolithic" image types have some hurd-specific configuration, and only "monolithic" can be built (due to missing udeb deps on "netboot", IIRC). [TODO: reword and complete: Once a basic installer is working, getting the same variety in the boot images as is available for Linux would be ...] Right now, only the "netboot" and "monolithic" image types have some hurd-specific configuration,
and only "monolithic" can be built (due to missing udeb deps on "netboot", IIRC).
Ultimately, more image types should be supported,
but this
would probably require some shuffling of the pkg-lists
to separate some system-specific parts
.
Line 94: Line 133:
Though it's somewhat sluggish, Xorg runs on Hurd, and porting the graphical installer would be great for languages not supported by the Hurd console. However I have not reviewed the situation in depth. Though it's somewhat sluggish, Xorg runs on Hurd,
and porting the graphical installer would be great for languages not supported by the Hurd console.

=== ... and many other cool surprises! ===

I expect to be disco
vering much more work as I progress.
But if we could know e
verything in advance, there would be no fun!
Line 98: Line 143:
[TODO]
 
With this in mind, here is my attempt at a schedule.
I'm confident that I would be able to achieve the first half of it
even if an unexpected number of unexpected problems show up...

||'''Timeframe''' ||'''Stuff being done'''||
|| Evaluation period || fix the bootstrap bug, maybe port init in busybox ||
|| Community bonding period || read some docs, try to get the installer starting ||
|||| ''At this point I would have tried to get through a complete installation with manual tweaks and see what breaks'' ||
|| Week 1 (2010-05-24) || warm up: polish what we have (busybox, genext2fs), setup xen ||
|| Week 2 (2010-05-31) || fix the easy stuff (install-time netcfg, cdrom detection, whatever) ||
|| Week 3 (2010-06-07) || Mach initrd, Hurd tweaks ||
|| Week 4 (2010-06-14) || partitionning, bootloader ||
|| Week 5 (2010-06-21) || unexpected stuff, cleanup ||
|| Week 6 (2010-06-28) || call for testing, fix the bugs ||
|||| ''At this point we would have a basic but working installer. Hopefully the summer would not be over yet...'' ||
|| Week 7 (2010-07-05) || Hurd console ||
|| Week 8 (2010-07-12) || more image types, better DHCP support ||
|| Week 9 (2010-07-19) || give a shot at the graphical installer ||
|| Week 10 (2010-07-26) || unexpected stuff, cleanup ||
|| Week 11 (2010-08-02) || shake the thing, call for testing, fix the bugs ||
|| Week 12 (2010-08-09) || fix the bugs, write documentation, fix the bugs ||
Line 110: Line 175:
I'm quite fond of Debian, both as an operating system and as an organization, so I hope that a participation to GSoC would be the foot in the door of my continued involvement :-) In any case, I expect to be able to maintain the code I would have produced. Obviously, even if the project is a success, lots of work would remain on Debian GNU/Hurd.
I'm quite fond of Debian, both as an operating system and as an organization,
and I have had other Debian-related projects (see my outdated [[http://jk.fr.eu.org/|website]]),
so I hope that a participation to GSoC would be the foot in the door of my continued involvement :-)

Title: Hurd Debian-Installer (draft)

Student: Jeremie Koenig <jk@jk.fr.eu.org>, jkoenig@{freenode,oftc}

Mentor: Samuel Thibault (youpi) <sthibault@debian.org>

Abstract: Debian GNU/Hurd is currently installed either using outdated CD images, or from an existing Debian GNU/Linux system using the 'crosshurd' package. The goal of this project is to modify debian-installer and the related packages to produce working Debian GNU/Hurd installation images.

Update 2010-05-24: The proposal has been accepted by Debian. I will keep track of my progress on the SummerOfCode2010/HurdDebianInstaller/JeremieKoenig/Roadmap page.

Background

I'm a 25 years old Computer Science student in Strasbourg (France), currently finishing my Licence (more or less equivalent to a bachelor's degree I think), after some years of interruption.

In particular I worked for some time as a developer on a Debian-based product (an "embedded" backup server with a web interface). This involved creating custom packages and modifying existing ones, automating the system's administration and installation, as well as some C, Perl, PHP and C# programming. Though the work in itself was fun and went rather smoothly, it was not manageable as a part-time job so I had to quit to resume my studies.

I have never been a regular Debian contributor, but I have used it for quite some time (see the list of bugs I reported) and am familiar with the developer tools. I have also done some debian-installer work in the past, namely helping with "oldworld" Macintosh support. I had to tweak the d-i image build scripts, package the miBoot bootloader and create the rsrce package (used to configure miBoot). Unfortunately, there were legal problems with miBoot and the Macintosh boot sector which the floppy used, so my work could not be completely integrated.

As for the Hurd I am less familiar with its guts, but I am enthusiastic about its overall design and I had tried it in the past. Preparing this application also involved some looking under the hood and tinkering with it, so I am confident that I would be able to acquire the necessary knowledge without too much of a hassle.

Project overview

Hurd is micro-kernel based system: most of the code usually found in the kernel of Unix-like systems is implemented in userspace, as a series of cooperating server processes. The filesystem is used as a namespace for the services that the system provides (the servers are said to translate parts of the filesystem), and the principle of capabilities is used to manage access to resources. This design offers great flexibility, as even a normal user can use to own servers, in addition to or as a replacement to the system ones.

I would work on a native debian-installer port for Hurd. Colin Watson and Samuel Thibault have already done some work in this direction, but much remains. Besides the installer itself, the project would involve at least ?BusyBox, GNU Mach and Hurd. Furthermore, I intend to coordinate with the kFreeBSD people and come up with generic solutions for non-Linux systems whenever possible.

Benefits to Debian

I believe that Debian, the universal operating system, benefits from non-Linux ports in general:

  • Its users get more choice and more functionnality (FreeBSD jails, KAME -- or the crucial random .signature translator)
  • The free software community gets the whole Debian archive built for alternative kernels, which is great to ensure that those packages are portable and that they target POSIX rather than Linux.

More specifically, a Hurd debian-installer would be an important step towards making hurd-i386 a viable Debian architecture (though, admittedly, lots of work would remain). Making debian-installer less tied to a particular kernel would benefit non-Linux Debian architectures in general.

Deliverables

My work would primarily consist of changes to existing software. I would submit them as I go, as patches against Debian packages and upstream where appropriate. I would also keep track of them, so I can submit them to Google after GSoC is over.

Project details

Current status

Samuel Thibault has some packages and patches which can be used to build "monolithic" debian-installer images, which at some point he could boot into a shell. I have fixed a bug in ext2fs which prevented them from booting but as of 2010-04-06, another problem with the bootstrap process remains (namely, it hangs after the exec server is started by ext2fs.)

Busybox

I have started working on "clean" patches for porting busybox. My first patches have been accepted upstream with some changes and the source from the upstream git repository builds on Hurd with a minimal configuration. However more applets will probably be needed as I progress with the installer.

In any case, some applets will need to be disabled in the Debian package on Hurd and kFreeBSD. This could be done either by using system-specific configuration files in the Debian package, or by modifying the build system of busybox: the current target system would be detected, and the irrelevant applets would be disabled through KConfig dependencies.

Genext2fs

The ext2fs translator only supports filesystems with 4KB blocks right now. On the other hand, genext2fs uses a fixed block size of 1024. The problem could be attacked on either side, but the genext2fs route seems much more simple. Samuel Thibault has a patch which changes the hardwired #define from 1024 to 4096, and sets the superblock's "creator OS" field to a system-specific value. In the longer term, however, these would need to be turned into command-line options.

GNU Mach

Mach will need to support using some of its multiboot modules as initial ramdisks. I have had an occasion to wander around the code for iopl and the "driver" part for this does not seem too hard. The user interface might be fitted into the boot script language, for instance a $(use-as-initrd) statement on a module's command line would trigger its usage as a ramdisk device.

The decompression could be done either by GRUB or by libstore. In either case the ramdisk could be used read-write.

Hurd

Crosshurd currently does some magic in the "native-install" script to set up translators on a newly installed system. This magic would need to be moved into the hurd package's postinst script so that it happens during d-i installations.

Tuning hurd-udeb and further changes might be necessary as well. For instance, there are currently workarounds for the installer in the Debian package's libstore bootstrap code and I strongly suspect that they will need some changes or cleaning up.

Network configuration

Apparently, netcfg has had Hurd support for some time (r548, by David Whedon starts it, the more recent r62649 by Samuel Thibault adds some fixes). However, I'm not sure that DHCP is supported, which it also isn't in a straightforward way on an installed Hurd system.

So the situation would need to be reviewed and support for configuring DHCP both during the install and on the target system would need to be implemented. For the record, Samuel Thibault has submitted a patch which adds Hurd support to dhcp3.

Partitioning

Not every filesystem can be used for a Hurd installation: support for passive translators is necessary. As a consequence, we have to use ext2 as the root filesystem, with special options at mkfs time as mentioned above for genext2fs. More generally, Hurd only has a subset of the filesystems which Linux understands. Partman would have to be modified to stay within these constraints on Hurd installations.

Bootloader installation

Both os-prober and grub-installer are Linux-specific for now, though there is some support for Hurd as a foreign OS. The Linux-specific parts would have to be moved into "hook" shell functions, and each system would provide a shell script snippet implementing them. debian/rules could then choose the relevant snippet at build time.

Multilingual console

Mach has a very basic console which could be used in the beginning.

Among other interesting features, the Hurd console has some UTF-8 support and its VGA driver can use any 512 glyphs at once in a dynamic fashion. So while I don't expect it to handle bidi or large glyphs, I believe it could be used for a fair number of languages nontheless.

The Hurd console uses BDF fonts, so a udeb similar to bterm-unifont would have to be created. Also, the keyboard driver does not support any kind of keymaps so this would need to be implemented, and integrated with kdb-chooser and /etc/default/keyboard-configuration.

Installation media

Right now, only the "netboot" and "monolithic" image types have some hurd-specific configuration, and only "monolithic" can be built (due to missing udeb deps on "netboot", IIRC). Ultimately, more image types should be supported, but this would probably require some shuffling of the pkg-lists to separate some system-specific parts.

Graphical installer

Though it's somewhat sluggish, Xorg runs on Hurd, and porting the graphical installer would be great for languages not supported by the Hurd console.

... and many other cool surprises!

I expect to be discovering much more work as I progress. But if we could know everything in advance, there would be no fun!

Project schedule

With this in mind, here is my attempt at a schedule. I'm confident that I would be able to achieve the first half of it even if an unexpected number of unexpected problems show up...

Timeframe

Stuff being done

Evaluation period

fix the bootstrap bug, maybe port init in busybox

Community bonding period

read some docs, try to get the installer starting

At this point I would have tried to get through a complete installation with manual tweaks and see what breaks

Week 1 (2010-05-24)

warm up: polish what we have (busybox, genext2fs), setup xen

Week 2 (2010-05-31)

fix the easy stuff (install-time netcfg, cdrom detection, whatever)

Week 3 (2010-06-07)

Mach initrd, Hurd tweaks

Week 4 (2010-06-14)

partitionning, bootloader

Week 5 (2010-06-21)

unexpected stuff, cleanup

Week 6 (2010-06-28)

call for testing, fix the bugs

At this point we would have a basic but working installer. Hopefully the summer would not be over yet...

Week 7 (2010-07-05)

Hurd console

Week 8 (2010-07-12)

more image types, better DHCP support

Week 9 (2010-07-19)

give a shot at the graphical installer

Week 10 (2010-07-26)

unexpected stuff, cleanup

Week 11 (2010-08-02)

shake the thing, call for testing, fix the bugs

Week 12 (2010-08-09)

fix the bugs, write documentation, fix the bugs

Travel

I expect to be able (and would be more than willing) to travel to Debconf 10. However I'm on a tight budget at the moment so I would need sponsoring for the travel and housing costs.

Other plans

I don't have any other other plans for this summer and I expect to be able to work on this project full-time for the complete duration of the GSoC program. My exams should be finished by May 21, so I don't expect them to be a problem either.

After GSoC

Obviously, even if the project is a success, lots of work would remain on Debian GNU/Hurd. I'm quite fond of Debian, both as an operating system and as an organization, and I have had other Debian-related projects (see my outdated website), so I hope that a participation to GSoC would be the foot in the door of my continued involvement :-)