Differences between revisions 43 and 45 (spanning 2 versions)
Revision 43 as of 2011-06-21 07:03:58
Size: 9648
Editor: RonnyAasen
Comment: another path name
Revision 45 as of 2011-09-13 06:24:19
Size: 9865
Editor: RonnyAasen
Comment: added 2 missing file names to the list of autogenerated files. and more readable
Deletions are marked like this. Additions are marked like this.
Line 120: Line 120:
 1. after the first build. you should add the files wantedpkglist-$suite filelist-$suite pkgdeblist-$suite missingpkglist-$suite cdspacelist-$suite to svn. If you don't you will not recive commit mails after the build.
 1. if you add a new version, you must also make sure there is a symlink to our local -test repo in the temp_files/mirror-link
 1. after the first build. you should add the autogenerated files files to svn. If you don't you will not recive commit mails after the build.
    * wantedpkglist-$suite.txt
    * unwantedpkglist-$suite.txt
    * filelist-$suite.txt
    * pkgdeblist-$suite.txt
    * missingpkglist-$suite.txt
    * cdspacelist-$suite.txt
    * sort-by-popcon-$suite.txt
 1. if you add a new version, you must also make sure there is a symlink to our local -test repo in the temp_files/mirror-link and a symlink to our local repo in the builder_temp_dir/local-mirror/dists

Any sufficiently advanced technology is indistinguishable from magic...

This is in the very early stage of something which should become a howto to understand the build process of the Debian Edu (install) cd - please help to improve it. I will do so too, on my way to eternal wisdom and happyness :-)

  • tasks comes from debian-edu svn (from the tasks dir, but somehow also the changelog version is used. Which h01ger doesnt understand, because the changelog says lenny-test but still, other images are build too)
  • wantedpkglist*.txt is generated from tasks, except for the netinstaller that does not have debs, there is is only generated from the hintfile-netinst.txt
  • unwantedpkglist*.txt is generated from debian-edu/tasks/common, look for the avoid section.
  • wantedpkglist is given to debian-cd to populate the cd
  • missingpkglist is generated by comparing the list of packages on the CD with the list of wanted packages.

someday someone might describe here whats needed to debug failing builds

<h01ger> white, pere, Werner, sep: can you give me hints how to debug the magic cd build?
<h01ger> i'm willing to update the docs :)
<white> well i guess becoming the builder user and then checking the Makefile in the ~builder/.../CD-administrator dir and running the build commands should eventually give you the error
<h01ger> how do i run the build commands?
<white> no idea to be honest, it should be in the Makefile, one of the commands there
<pere> h01ger: setting some environment variable should give you a build with output to stdout.
<pere> you have to check the scripts to find it.  possibly build.sh.
<white> you can also debug it by just running the cronjob and checking what the script is doing via the process list
<pere> the build log is copied to ~ftp/cd-*/ now, so you could have a look.
<pere> crontab -l to find the build job, check that script, see what it calls, check that script, etc..

If you want to test a build. you can run the following command. The suite sets what arch to test build. It is not certain this breaks the real build, but if possible try to avoid running them simultaneously

suite=lenny-test-i386 nice make -s -C $HOME/src/debian-edu/src/build/CD-administrator build-test

debian-cd

We are using two different branches of debian-cd. The svn://svn.debian.org/debian-cd/etch_r0_build branch is used to build etch cd's. And svn://svn.debian.org/debian-cd/trunk is used to build post-etch cd's. The reason for this is, that when debian release a revision update, they update the etch branch and we'd have to backport those updates in our patch if we wanted to build etch using trunk.

the upstream code for the version of debian-cd we're using is at /home/builder/src/debian-edu/src/build/CD-administrator/debian-cd.[etch|trunk] The upstream code is updated from svn by the patch-debian-cd script and uses the revision number found in the debian-cd-revision file. This makes it easier to merge in ustream changes when we have time to update our patch. And also easier to move back incase of severe breakage.

Since we use 2 branches we also have 2 patch files. debian-cd.patch.[etch|trunk]

During every build the matching upstream branch is patched by debian-edu/src/build/CD-administrator/patch-debian-cd <$branch> with debian-edu/src/build/CD-administrator/debian-cd.patch.$branch the result is put into debian-cd, where the build is executed.

If you want to exclude certain packages from the cd, there is a exclude-etch list (also used for etch-test) in that patch.

requirements on administrator.skoleliinux.no

administrator.skolelinux.no is the machine running the build, for each arch/image there are some needed prerequisites.

  • login as user builder via ssh. If you cant, ask drift to add your ssh-pub-key to builders authorized_keys file.

  • there need to be a temporary storage. as builder make a directory in /skolelinux/administrator/debmirror/builder_temp_dir that's called $suite eg: lenny-test-amd64

  • there needs to be a place to save the resulting image. make a directory in /var/www/ftp.skolelinux.org that's called cd-@suite eg: cd-lenny-test-amd64, this should be owner builder, and group www, and sticky group.
    • make sure this is also added to the rsync.conf

The mirror link or the local mirror

In order to avoid having to deal with local vs local-test in d-e-i and d-e-c (Speculation) We always add the local* repo as local on the images. This is done with a mirror-link directory in the build temp area. The mirror link directory contains a pool, that is taken from lenny-test and a dist directory containing symlinks to lenny-test dist, but with -test stripped off. This way we can build images with different repo's on them but they are allways called "local" on the image.

  • example : lenny -> /var/www/ftp.skolelinux.org/skolelinux/dists/lenny-test

cd-build logs

the logs from each cd build are commited to svn (debian-edu/html/logs/cd-build-$suite.log). the commits are silent, so no commit-mail is sent. You can read the logs in the individual cd-suite directories in http://ftp.skolelinux.no

force re-build

all developers can force a build by using the command /home/builder/bin/trigger_builds on user.skolelinux.org - trigger_build takes on argument: the image to rebuild or "all".

how to exclude packages from the cd

Get the curret status

  1. grab CD-administrator with svn co svn+ssh://$alioth-user-name@svn.debian.org/svn/debian-edu/trunk/src/build/CD-administrator

  2. cd CD-administrator
  3. patch using ./patch-debian-cd

edit the patch

  • /!\ make-patch is not working at the moment

  • edit the exlude file with vi debian-cd/tasks/exclude-etch

  • regenerate the patch with ./make-patch, this will generate a debian-cd.patch file.

  • Test that your patch works with ./patch-debian-cd $branch

  • commit your changes to the patchfile with svn commit debian-cd.patch

  • have a beer and wait for the cd to build

what we are patching and why

FIXME: In here we should try to document what our patch actualy does. Kind of like a specification for our requirements for debian-cd

  • debian-cd/build.sh

FIXME: add rationale.

  • debian-cd/data/lenny/${ARCHES}_udeb_include

We patch a whole lot of these files. basically we add our udeb named debian-edu-install-udeb, this ends up on the cd in the file .disk/udeb_include and is used by d-i to know what udebs to load.

  • debian-cd/tasks/lenny/Debian-generic and debian-cd/tasks/lenny/forcd1

We patch Debian-generic to make debian-cd include the packages referenced in our file debian-edu, i suspect later versions of debian-cd, uses Debian-* task files depending on what desktop is built, we are building -kde. so we adding it to forcd1 to enforce our packages are on cd1. Notice that the file debian-edu is missing, it's copied from wantedpkglist-$suite.txt at build time by our build.sh script.

  • debian-cd/Makefile

A hack to make a source cd exactly from whats on our binary DVD

  • debian-cd/tasks/lenny/exclude

Packages we do not want on our CD, same as with tasks/$codename/debian-edu this is copied into debian-cd by our build.sh script.

  • debian-cd/tools/add-bin-doc

FIXME: add rationale

  • debian-cd/tools/boot/lenny/boot-x86

  • FIXME: add rationale
  • Adds our own splash logo ;; This can probably be replaced by the new SPLASHPNG config variable.
  • debian-cd/tools/sort_deps

FIXME: add rationale

how to merge in changes from upstream

  1. increment debian-cd-revision, svn
  2. patch all the variants using ./patch-debian-cd [lenny|squeeze|nolocal|trunk|*]

  3. if all patches succeeds, you can commit debian-cd-revision, and wait for the cd to build.
  4. if one (or more) of the patches fail, you'll have the half baked results in debian-cd.new. Check thru the rejects and adjust debian-cd.patch.VARIANT as needed. Test that your modified patch works with ./patch-debian-cd VARIANT again

  5. commit the debian-cd-revision, and debian-cd.patch.$branch files
  6. have a $beverage while it's building.
  7. test the cd

How to build more images

There are a few thing one must in order to add a new cd to the build system.

  1. make the temp file area.
    • debian-cd uses a area on disk, for tempfile area, building the filesystem that becomes the iso, and making the iso.

      you need to make the directory owned by builder as /skolelinux/administrator/debmirror/builder_temp_dir/$suite. For example /skolelinux/administrator/debmirror/builder_temp_dir/lenny-test-dvd/  You should also check that the area have the needed free space 1.5G for a cd and FIXME: GB for the dvd

  2. Make a new config file called CONF-$suite, eg CONF-lenny-test-dvd.sh, and edit it to suite your needs
  3. add the suite to the Makefile, in the suites= list
  4. add the suite to the trigger tool, located in builder's ~/bin/trigger_builds
  5. after the first build. you should add the autogenerated files files to svn. If you don't you will not recive commit mails after the build.
    • wantedpkglist-$suite.txt
    • unwantedpkglist-$suite.txt
    • filelist-$suite.txt
    • pkgdeblist-$suite.txt
    • missingpkglist-$suite.txt
    • cdspacelist-$suite.txt
    • sort-by-popcon-$suite.txt
  6. if you add a new version, you must also make sure there is a symlink to our local -test repo in the temp_files/mirror-link and a symlink to our local repo in the builder_temp_dir/local-mirror/dists

buildd admins

Members of the builder group on administrator are:

  • Petter
  • Steffen
  • Ronny

famous last words

... and any technology distinguishable from magic is not sufficiently advanced!