Differences between revisions 35 and 37 (spanning 2 versions)
Revision 35 as of 2009-03-16 22:28:56
Size: 8797
Editor: RonnyAasen
Revision 37 as of 2009-11-26 12:27:48
Size: 9015
Editor: HolgerLevsen
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
 * unwantedpkglist*.txt is generated from debian-edu/tasks/common, look for the avoid section.
Line 44: Line 45:
 * login as user '''builder''' via ssh. If you cant, ask drift to add your ssh-pub-key to builders authorized_keys file.

Any sufficiently advanced technology is indistinguishable from magic...

This is in the very early stage of something which should become a howto to understand the build process of the Debian Edu (install) cd - please help to improve it. I will do so too, on my way to eternal wisdom and happyness :-)

  • tasks comes from debian-edu svn (from the tasks dir, but somehow also the changelog version is used. Which h01ger doesnt understand, because the changelog says lenny-test but still, other images are build too)
  • wantedpkglist*.txt is generated from tasks, except for the netinstaller that does not have debs, there is is only generated from the hintfile-netinst.txt
  • unwantedpkglist*.txt is generated from debian-edu/tasks/common, look for the avoid section.
  • wantedpkglist is given to debian-cd to populate the cd
  • missingpkglist is generated by comparing the list of packages on the CD with the list of wanted packages.

someday someone might describe here whats needed to debug failing builds

<h01ger> white, pere, Werner, sep: can you give me hints how to debug the magic cd build?
<h01ger> i'm willing to update the docs :)
<white> well i guess becoming the builder user and then checking the Makefile in the ~builder/.../CD-administrator dir and running the build commands should eventually give you the error
<h01ger> how do i run the build commands?
<white> no idea to be honest, it should be in the Makefile, one of the commands there
<pere> h01ger: setting some environment variable should give you a build with output to stdout.
<pere> you have to check the scripts to find it.  possibly build.sh.
<white> you can also debug it by just running the cronjob and checking what the script is doing via the process list
<pere> the build log is copied to ~ftp/cd-*/ now, so you could have a look.
<pere> crontab -l to find the build job, check that script, see what it calls, check that script, etc..

If you want to test a build. you can run the following command. The suite sets what arch to test build. It is not certain this breaks the real build, but if possible try to avoid running them simultaneously

suite=lenny-test-i386 nice make -s -C $HOME/src/debian-edu/src/build/CD-administrator build-test


We are using two different branches of debian-cd. The svn://svn.debian.org/debian-cd/etch_r0_build branch is used to build etch cd's. And svn://svn.debian.org/debian-cd/trunk is used to build post-etch cd's. The reason for this is, that when debian release a revision update, they update the etch branch and we'd have to backport those updates in our patch if we wanted to build etch using trunk.

the upstream code for the version of debian-cd we're using is at /home/builder/src/debian-edu/src/build/CD-administrator/debian-cd.[etch|trunk] The upstream code is updated from svn by the patch-debian-cd script and uses the revision number found in the debian-cd-revision file. This makes it easier to merge in ustream changes when we have time to update our patch. And also easier to move back incase of severe breakage.

Since we use 2 branches we also have 2 patch files. debian-cd.patch.[etch|trunk]

During every build the matching upstream branch is patched by debian-edu/src/build/CD-administrator/patch-debian-cd <$branch> with debian-edu/src/build/CD-administrator/debian-cd.patch.$branch the result is put into debian-cd, where the build is executed.

If you want to exclude certain packages from the cd, there is a exclude-etch list (also used for etch-test) in that patch.

requirements on administrator.skoleliinux.no

administrator.skolelinux.no is the machine running the build, for each arch/image there are some needed prerequisites.

  • login as user builder via ssh. If you cant, ask drift to add your ssh-pub-key to builders authorized_keys file.

  • there need to be a temporary storage. as builder make a directory in /skolelinux/administrator/temp_storage/builder/ that's called $suite eg: lenny-test-amd64
  • there needs to be a place to save the resulting image. make a directory in /var/www/ftp.skolelinux.org that's called cd-@suite eg: cd-lenny-test-amd64, this should be owner builder, and group www, and sticky group.
    • make sure this is also added to the rsync.conf

cd-build logs

the logs from each cd build are commited to svn (debian-edu/html/logs/cd-build-$suite.log). the commits are silent, so no commit-mail is sent. You can read the logs in the individual cd-suite directories in http://ftp.skolelinux.no

force re-build

all developers can force a build by using the command /home/builder/bin/trigger_builds on user.skolelinux.org - trigger_build takes on argument: the image to rebuild or "all".

how to exclude packages from the cd

Get the curret status

  1. grab CD-administrator with svn co svn+ssh://$alioth-user-name@svn.debian.org/svn/debian-edu/trunk/src/build/CD-administrator

  2. cd CD-administrator
  3. patch using ./patch-debian-cd

edit the patch

  • /!\ make-patch is not working at the moment

  • edit the exlude file with vi debian-cd/tasks/exclude-etch

  • regenerate the patch with ./make-patch, this will generate a debian-cd.patch file.

  • Test that your patch works with ./patch-debian-cd $branch

  • commit your changes to the patchfile with svn commit debian-cd.patch

  • have a beer and wait for the cd to build

what we are patching and why

FIXME: In here we should try to document what our patch actualy does. Kind of like a specification for our requirements for debian-cd

  • debian-cd/build.sh

FIXME: add rationale.

  • debian-cd/data/lenny/${ARCHES}_udeb_include

We patch a whole lot of these files. basically we add our udeb named debian-edu-install-udeb, this ends up on the cd in the file .disk/udeb_include and is used by d-i to know what udebs to load.

  • debian-cd/tasks/lenny/Debian-generic and debian-cd/tasks/lenny/forcd1

We patch Debian-generic to make debian-cd include the packages referenced in our file debian-edu, i suspect later versions of debian-cd, uses Debian-* task files depending on what desktop is built, we are building -kde. so we adding it to forcd1 to enforce our packages are on cd1. Notice that the file debian-edu is missing, it's copied from wantedpkglist-$suite.txt at build time by our build.sh script.

  • debian-cd/Makefile

A hack to make a source cd exactly from whats on our binary DVD

  • debian-cd/tasks/lenny/exclude

Packages we do not want on our CD, same as with tasks/$codename/debian-edu this is copied into debian-cd by our build.sh script.

  • debian-cd/tools/add-bin-doc

FIXME: add rationale

  • debian-cd/tools/boot/lenny/boot-x86

  • FIXME: add rationale
  • Adds our own splash logo ;; This can probably be replaced by the new SPLASHPNG config variable.
  • debian-cd/tools/sort_deps

FIXME: add rationale

how to merge in changes from upstream

  1. increment debian-cd-revision, svn
  2. patch both the trunk and the etch branch using using ./patch-debian-cd [etch|trunk]

  3. if both patches succeeds, you can commit debian-cd-revision, and wait for the cd to build.
  4. if one (or both) of the patches fail, you'll have the half baked results in debian-cd.new. Check thru the rejects and adjust debian-cd.patch.[etch|trunk] as needed. Test that your modified patch works with ./patch-debian-cd [etch|trunk]

  5. commit the debian-cd-revision, and debian-cd.patch.$branch files
  6. have a $beverage while it's building.
  7. test the cd

How to build more images

There are a few thing one must in order to add a new cd to the build system.

  1. make the temp file area.
    • debian-cd uses a area on disk, for tempfile area, building the filesystem that becomes the iso, and making the iso.

      you need to make the directory owned by builder as /skolelinux/administrator/temp_storage/builder/$suite. For example /skolelinux/administrator/temp_storage/builder/lenny-test-dvd/  You should also check that the area have the needed free space 1.5G for a cd and FIXME: GB for the dvd

  2. Make a new config file called CONF-$suite, eg CONF-lenny-test-dvd.sh, and edit it to suite your needs
  3. add the suite to the Makefile, in the suites= list
  4. add the suite to the trigger tool, located in builder's ~/bin/trigger_builds
  5. after the first build. you should add the files wantedpkglist-$suite filelist-$suite pkgdeblist-$suite missingpkglist-$suite cdspacelist-$suite to svn. If you don't you will not recive commit mails after the build.
  6. if you add a new version, you must also make sure there is a symlink to our local -test repo in the temp_files/mirror-link

buildd admins

Members of the builder group on administrator are:

  • Petter
  • Steffen
  • Ronny

famous last words

... and any technology distinguishable from magic is not sufficiently advanced!