Differences between revisions 1 and 124 (spanning 123 versions)
Revision 1 as of 2006-03-04 20:54:49
Size: 72
Editor: ?SteffenJoeris
Comment: initial release
Revision 124 as of 2010-06-13 10:34:14
Size: 20341
Editor: HolgerLevsen
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
            ftpmaster-Howto
-----------------------------------------
<<TableOfContents(3)>>

= ftpmaster-Howto =

The dak system is located on administrator.skolelinux.no - see the [[http://ftp.skolelinux.org/skolelinux/needs_love.html|current status]] here, and the [[http://ftp.skolelinux.org/skolelinux/new.html|new queue]] there.

After you logged in on a.s.n. you have to become the katie user. If you are a ftpmaster you can use the command "sudo -u katie bash" alternativly use "sudo -i -u katie" this sets shell and home automatically. Now you are the debadmin user and have access to the archive system.

== dak upgrade, changes to document ==

 * /opt/dak/katie/katie.conf has been moved to /opt/dak/config/dak.conf
 * lots of other stuff I dont know
 * {{{dak clean-suites}}} fails, probably because of DM changes to dak
 * {{{dak rm -h}}} is useful to know as well
 * dak repo: http://ftp-master.debian.org/bzr/ftpmaster-dak

== Configuration: ==
If it is neccessary to change a config file or a script under bin, cron or katie you have to change them first and then call "cvs diff" and "cvs commit".


== NEW Handling: ==
One of the most important jobs of the ftpmaster is to handle the NEW queue. The NEW queue are packages which are signed by a Debian-Edu Developer and passed the key check. So you don't need to check if the signature is valid or not.

The ftpmaster's task is to check the package if it is ready for Debian-Edu or not. To get a checklist you can use the Debian-Edu Archive Policy or [[http://ftp-master.debian.org/REJECT-FAQ.html|the Debian reject-FAQ]] and check if it passes all points. Other important checks can be lintian, or debdiff against the package in debian proper.

You should check that $HOME is set correctly or set it with {{{export HOME=/var/lib/dak}}}
Then change into the directory /org/ftp.skolelinux.org/queue/new/ There you find the packages.
After you have checked the package you have to run {{{ dak process-new package_xxx-x_arch.changes }}}

Then you can type "c" and check some important points of the packages again. After you made your decision you can either accept or reject the package.

To accept it as it is (means no changes to maintainers suggestion for override) just type "e" and change the "Section" into "local/section" where "section" is the old one, e.g. admin, debian-installer, misc, ... . Then go back, using d (Done). You will see the Add overrides opportunity and you can type "a" . Then the override file is changed. If you disagree to the maintainer and e.g. want to change the section field you can run "e". Then make your changes and the package is accepted.

If you want to reject the package type "m". Then you get a text editor (vim) and can describe the reason there.
In case of rejection please use the following template form:
{{{
Hi Maintainer, ...

... reason ...
rejected for now.
}}}

At the end of the rejection message lisa will automatically insert the text from the template which can be found under "/org/ftp.skolelinux.org/katie/templates/katie.rejected"
Please do not edit the template.

If you accidently rejected a package, you can just move all files (except *.reason) from ./reject to ./unchecked.

== Moving packages to other distributions ==
Packages are usualy requested to be moved into the stable (non -test) by a developer. You can use the status pages for [[http://wiki.debian.org/DebianEdu/Status/Etch|Etch]] and [[http://wiki.debian.org/DebianEdu/Status/Lenny|Lenny]] and compare with the dynamicaly generated page at http://ftp.skolelinux.org/skolelinux/etch_needs_love.html

Things to look at can be
 * Read the changelog about the changes from the stable package to the proposed package
 * debdiff stablepackage.deb proposedpackage.deb | diffstat
 * debdiff stablepackage.deb proposedpackage.deb | less
 * do NOT move a package, that you yourself have changes in, since last version in stable package.

Packages should be uploaded to '''lenny-test''' for now. After they have been there for a while, and the maintainer and/or ftpmaster deceided to move it to '''lenny''', the ftpmasters need to move them '''by hand'''. This is to make sure only stable and good packages end up in the '''stable''' repository. Packages are moved from '''sarge-test''' to '''sarge''', '''etch-test''' to '''etch''' or from '''lenny-test''' to '''lenny'''. The command to be executed is as follows
{{{
/org/ftp.skolelinux.org/bin/package-sync -s lenny-test package-name
}}}

This command will move '''package-name''' from etch-test to etch.

/!\ Currently etch is '''untouchable''' (=frozen) and this command (alone) doesnt work. See below.

''Don't wory about packages that are uploaded with distribution '''etch''' in their changelog, they will end up in '''etch-test''' first. :-)''

package-name is the name of the source package. package-sync then moves all binary packages built from this source package.

=== when $suite is untouchable ===
 1. prepare the announcement mail in svn under stable-updates/package.codename.version
 1. if HOME is set incorrectly: {{{export HOME=/var/lib/dak ; export LANG=C}}}
 1. deactivate the cronjobs, {{{vi /etc/cron.d/dak}}}
 1. then copy the dak.conf file away: {{{cp /etc/dak/dak.conf /etc/dak/dak.conf.bak}}}
 1. then remove the "Untouchable" line for the etch suite in katie.conf, {{{vi /etc/dak/dak.conf}}}
 1. then move the package as usual (after checking, etc.)
 1. then run dinstall: {{{/opt/dak/cron/cron.dinstall}}}
 1. then copy the old dak.conf back (with the untouchable line), {{{mv /etc/dak/dak.conf.bak /etc/dak/dak.conf}}}
 1. check the archive by running {{{dak ls $package}}}
 1. first do {{{touch /org/ftp.skolelinux.org/RUN-DINSTALL}}} and run dinstall again {{{/opt/dak/cron/cron.dinstall}}}
 1. and activate the cronjobs again, {{{vi /etc/cron.d/dak}}}
 1. and then check again with dak ls and maybe run dinstall a third time
 1. send the announce mail, gpg signed, to debian-edu-announce@lists.debian.org
 1. update the stable status page, by moving the package from proposed updates to, updated packages on the [[http://wiki.debian.org/DebianEdu/Status/Etch|Stable status page]]

=== when $package is out of sync on different archs ===

_Sometimes_ its useful to migrate a package on one arch only, because the others have not been build yet. In general this should be avoided, but if you have to do it, here is how to clean up the mess:

/!\ As with everything here: only run the commands if you understand what they do!

{{{
 dak ls debian-edu
 dak ls education-standalone
 dak ls -a powerpc -s lenny-test debian-edu -S -f heidi
 dak ls -a powerpc -s lenny-test debian-edu -S -f heidi | dak control-suite -a lenny
 dak ls -a amd64 -s lenny-test debian-edu -S -f heidi
 dak ls -a amd64 -s lenny-test debian-edu -S -f heidi | dak control-suite -a lenny
 dak ls debian-edu
 dak ls education-standalone
}}}

== removing packages ==

to remove packages from the archive: dak rm -s $suite -C debian-edu@lists.debian.org $sourcepackage

about the other options: dak rm -h :)

to remove packages from queue/unchecked (for example, because they were rejected, because the key is not in the keyring): just rm them!

If you want/need to use rene, dont use rene in path but the one in /org/ftp.skolelinux.org/katie/bin/ /!\ FIXME: path does not exist.

Steffen also told us on IRC that we should remove the appropriate .changes file from the done-directory, then run a dinstall and then probably remove the files manually after removing a package with the dak commands. And in the end we of course should check everything :)

If you need to do a complete removal, http://folk.uio.no/werner/ftpmaster_remove_slbackup-php.html has a transcript of a complete removal session.

== un-removing packages ==

If you removed a package with "dak rm" you can un-remove it like this:

{{{
echo flashplugin-nonfree-extrasound 0.0.svn2431-2.0.edu.etch.2 source | dak control-suite -a etch-test
echo flashplugin-nonfree-extrasound 0.0.svn2431-2.0.edu.etch.2 i386 | dak control-suite -a etch-test
}}}

You need to specify each arch and suite.

== New Accounts: ==
Please note that only drift@skolelinux.org decides to give someone account rights! [[http://wiki.debian.org/DebianEdu/NewContributor|New contributors]] should read this first.
If drift writes a mail to ftpmasters and tells them that a new Uploader needs an account for upload rights the ftpmaster need the gpg-key from the new Uploader.
For that purpose please write the new Uploader direclty and ask after a mail or get the key id and fingerprint from drift@skolelinux.org.

If the ftpmasters are instructed to create an account they can follow this procedure:

After login on administrator.skolelinux.no they have to become katie user ("sudo -u katie bash"). After that they should set the environment variable HOME by typing "export HOME=/var/lib/dak" .
Then they can use the command: {{{/org/ftp.skolelinux.org/bin/new-account -k ''keyid''}}}

Now you have to whitelist the uploaders email-address: checkout dak-config via cvs from /org/ftp.skolelinux.org/cvsroot and edit mail-whitelist and commit it with "cvs ci mail-whitelist". Now cd to /org/ftp.skolelinux.org/katie/ and do "cvs up".
Please make sure that you use an up to date cvs working copy before editing the file, to prevent conflicts at commit.

After that the new account is ready to be used and the new Uploader will get a mail which informs about it and explains some configuration steps for dput/dupload. Please add the name of the Uploader to the Uploaders-list which can be found under:
http://wiki.debian.org/DebianEdu/Uploaders

== Adding new keys to old accounts ==

as user kathie run:
{{{
export HOME=/var/lib/dak ; export LANG=C
gpg --keyserver pgpkeys.pca.dfn.de --no-default-keyring --keyring /opt/dak/keyrings/keyring.gpg --recv-keys E63CD6D6
}}}
to add new subkeys for the existing key ''E63CD6D&'' in the keyring.

== What does each script do ==

Scripts
 * Katie is just de username under which these scripts run, also see /org/ftp.sk.../cron/
 * melanie: removes packages

 * dak ls - shows which suites/archs a package is in
 * dak control-suite - list packages in suite and moves/copies packages between suites
 * new-account Specific to Debian-edu
      adds user to keyring and thus gives upload rights for edu-DDs
  Replaces Emilie in this setup
  Restricted access (peter and bana)
  Usage:
  set the $HOME variable to /var/lib/dak, then `sabina -k keyid` -> sends email with info on upload rights and conf
 * Jennifer run by /cron/unchecked, does the checks for ../unchecked
 * Helena reports by email what's in new or `helena -n ` gives html output of new (similar to the NEW normal debian thingy) - helena is also run by katies cronjobs
 * lisa is doing new processing and needs to be run manually (after helena mails)
  (cd $new ; lisa *.changes)
 * kelly - installs packages into the pool
 * jenna - generates lists of files in suites which are then fed to apt-ftparchive
 * ziyi - creates Release files
 * rhona + shania (ignore for now, cleanup stuff)
 * rene: sometimes needs to be run manually to check for obsolete packages (run daily via cron)
 * pgdump (backup the database)
 * Rose - Directory creation
  reads katie.conf and apt.conf
  used to add more suits
 * Neve - Database creation
  only at installation, it drops dtabase (postgresql)
  super user privileges (from stdin to databases)
 * Billy (not yet needed)
 * Emilie (not used in Debian-edu)
       GPG signatures checking and put the in the DB (use restricted)
 * Britney is for testing transition in Debian archive (not used in Debian-edu)

(We will upgrade to the new dak version without those scripts after the etch release.)

== the projectb database ==

Although the examples shown here are made as the katie user they should work as well with your user account. But this is only true for these examples, as only katie has write access.

=== using psql ===
{{{
katie@administrator:~$ psql projectb
Welcome to psql 7.4.19, the PostgreSQL interactive terminal.

Type: \copyright for distribution terms
       \h for help with SQL commands
       \? for help on internal slash commands
       \g or terminate with semicolon to execute query
       \q to quit

projectb=>

}}}

Now you can use normal sql queries to access dak's {{{projectb}}} database.

Entering {{{\z}}} at the psql prompt will list all tables with their access privileges settings, {{{\q}}} quits psql.

This explains how to learn more about the layout of the database:

{{{
projectb=> select * from pg_tables where schemaname='public';
 schemaname | tablename | tableowner | hasindexes | hasrules | hastriggers
------------+---------------------+------------+------------+----------+-------------
 public | architecture | katie | t | f | t
 public | archive | katie | t | f | t
 public | bin_associations | katie | t | f | t
 public | binaries | katie | t | f | t
 public | component | katie | t | f | t
 public | dsc_files | katie | t | f | t
 public | files | katie | t | f | t
 public | fingerprint | katie | t | f | t
 public | location | katie | t | f | t
 public | maintainer | katie | t | f | t
 public | override | katie | t | f | t
 public | override_type | katie | t | f | t
 public | priority | katie | t | f | t
 public | section | katie | t | f | t
 public | source | katie | t | f | t
 public | src_associations | katie | t | f | t
 public | suite | katie | t | f | t
 public | suite_architectures | katie | t | f | t
 public | accepted_autobuild | katie | f | f | t
 public | uid | katie | t | f | t
 public | queue | katie | t | f | f
 public | queue_build | katie | f | f | f
(22 rows)

projectb=> select * from pg_tables;
projectb=> select * from suite limit 1;
 id | suite_name | version | origin | label | policy_engine | description
----+------------+---------+--------------------+-------+---------------+--------------------
  5 | lenny | | Skolelinux archive | | | Skolelinux archive
(1 row)

projectb=>

}}}

=== using pg_dump ===

{{{pg_dump projectb}}} conviniently piped into {{{grep}}} allows to easily query the database on the commandline. An example says as much as 1000 words:

{{{
katie@administrator:~$ pg_dump projectb | grep multiseat
1825 local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3_amd64.deb 20810 1a6229f78968adfb4d811db7759fc252 2 \N
1826 local/m/multiseat/multiseat-udeb_0.9.10-0.0.edu.etch.3_amd64.udeb 11564 ccb66f75c1afe08d0a9c94080aa9253d 2 \N
699 local/m/multiseat/multiseat_0.9.10.orig.tar.gz 32092 081ff6a0475b1fd44e41975059f383de 2 \N
1476 local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3_powerpc.deb 20868 5abf45a8fddc0238d6919a99fd0664f4 2 \N
1477 local/m/multiseat/multiseat-udeb_0.9.10-0.0.edu.etch.3_powerpc.udeb 11554 cc89256ecb45fd9df8992dad03383c96 2 \N
717 local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3.dsc 613 b5de06c2af35ded51d7d333ec026b398 2 \N
718 local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3.diff.gz 3365 c11aaf4c17903fb79640e9f21cd38f77 2 \N
719 local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3_i386.deb 20512 77fa29628e752c2dcbbd3ce3d9da45d5 2 \N
720 local/m/multiseat/multiseat-udeb_0.9.10-0.0.edu.etch.3_i386.udeb 11340 bbbabdfe5a3fb8e66420f0e5a18b8ee6 2 \N
118 multiseat 0.9.10-0.0.edu.etch.3 4 717 2006-08-17 00:00:00 2
1255 multiseat 0.9.10-0.0.edu.etch.3 4 118 5 1825 deb 8
1256 multiseat-udeb 0.9.10-0.0.edu.etch.3 4 118 5 1826 udeb 8
466 multiseat 0.9.10-0.0.edu.etch.3 4 118 3 719 deb 2
467 multiseat-udeb 0.9.10-0.0.edu.etch.3 4 118 3 720 udeb 2
984 multiseat 0.9.10-0.0.edu.etch.3 4 118 4 1476 deb 7
985 multiseat-udeb 0.9.10-0.0.edu.etch.3 4 118 4 1477 udeb 7
multiseat 4 2 10 68 4 \N
multiseat-udeb 4 2 9 38 5 \N
katie@administrator:~$
}}}

== dak directory layout ==

 /org/ftp.skole..../queue
    /unchecked -> public incoming dir, cronjob
           (/cron/unchecked) looks at the changes file -> to new, accepted, rejected...
    /new
    /accepeted -> REPORT -> dinstall report
    /rejected
    /done -> after dinstall runs, changes files are here, the ret of the package goes to the pool
    
    -> REPORT -> file with logs on what happened
    


== Setup ==

dinstall is run every 15min., looks for changes for in /accepted or (the manually to be created file /org/ftp.skolelinux.org/RUN-DINSTALL), then it creates a lockfile

=== adding a new suite ===

 * edit /etc/dak/dak.conf as appropriate
 * edit /etc/dak/apt.conf as appropriate
 * run
{{{
touch /org/ftp.skolelinux.org/scripts/override/override.squeeze.local.debian-installer
touch /org/ftp.skolelinux.org/scripts/override/override.squeeze.local
mkdir -p /org/ftp.skolelinux.org/ftp/dists/squeeze-test/local
cd /org/ftp.skolelinux.org/ftp/dists/squeeze-test/local
mkdir binary-amd64 binary-i386 binary-powerpc debian-installer source
cd debian-installer/
mkdir binary-amd64 binary-i386 binary-powerpc
}}}
 * run {{{psql projectb}}} then
{{{
projectb=> INSERT INTO suite VALUES (DEFAULT,'squeeze-test','','Skolelinux archive','','','Skolelinux archive');
}}}

=== adding a new section ===

 * edit /etc/dak/dak.conf as appropriate
 * run {{{psql projectb}}} then
{{{
projectb=> INSERT INTO suite VALUES (DEFAULT,'local/debug');
}}}


=== you can't touch etch ===

After a release, suites are marked as "untouchable". In order to mark a suite untouchable, one of the ftpmasters adds the line "Untouchable "1";" to the end of the suite configuration in /org/ftp.skolelinux.org/katie/katie.conf . This tells dinstall to avoid touching the suite. If there is a need for a newer package, just uncomment the line from katie.conf and let the dinstall run.

== TODO-List for ftpmasters: ==

 1. there are no mails send for uploads to lenny*
 1. Finish documenting the new dak commands in the ftpmaster-howto.
 1. Put some of the dak stuff under a VCS.
 1. Contact the buildd admins to add the new suites to their system and check the scripts, which are run after dinstall (they generate the package overview page and trigger wanna-build)
 1. needs_love should be installed as katie, not holger
 1. needs_love now compares our packages against those on security.debian.org, should it send a mail in case something is newer there?
 1. run make_summary from a svn checkout
 1. we need to implement the changes into package-sync so that also a specific package version can be choosen
 1. document execution-stay-off-time
 1. document installation/configuration on a.s.n
 1. pere asks: why is the email about the etch update listing one of the binary packages and not the source package? For example education-astronomy - can we change this with reasonable effort?
 1. upgrade to latest dak after etch release
   * we manually fixed /usr/bin/melanie in line 165 (python syntax bug)
 1. sign the md5sum files of the cd/dvd images

ftpmaster-Howto

The dak system is located on administrator.skolelinux.no - see the current status here, and the new queue there.

After you logged in on a.s.n. you have to become the katie user. If you are a ftpmaster you can use the command "sudo -u katie bash" alternativly use "sudo -i -u katie" this sets shell and home automatically. Now you are the debadmin user and have access to the archive system.

dak upgrade, changes to document

  • /opt/dak/katie/katie.conf has been moved to /opt/dak/config/dak.conf
  • lots of other stuff I dont know
  • dak clean-suites fails, probably because of DM changes to dak

  • dak rm -h is useful to know as well

  • dak repo: http://ftp-master.debian.org/bzr/ftpmaster-dak

Configuration:

If it is neccessary to change a config file or a script under bin, cron or katie you have to change them first and then call "cvs diff" and "cvs commit".

NEW Handling:

One of the most important jobs of the ftpmaster is to handle the NEW queue. The NEW queue are packages which are signed by a Debian-Edu Developer and passed the key check. So you don't need to check if the signature is valid or not.

The ftpmaster's task is to check the package if it is ready for Debian-Edu or not. To get a checklist you can use the Debian-Edu Archive Policy or the Debian reject-FAQ and check if it passes all points. Other important checks can be lintian, or debdiff against the package in debian proper.

You should check that $HOME is set correctly or set it with export HOME=/var/lib/dak Then change into the directory /org/ftp.skolelinux.org/queue/new/ There you find the packages. After you have checked the package you have to run  dak process-new package_xxx-x_arch.changes 

Then you can type "c" and check some important points of the packages again. After you made your decision you can either accept or reject the package.

To accept it as it is (means no changes to maintainers suggestion for override) just type "e" and change the "Section" into "local/section" where "section" is the old one, e.g. admin, debian-installer, misc, ... . Then go back, using d (Done). You will see the Add overrides opportunity and you can type "a" . Then the override file is changed. If you disagree to the maintainer and e.g. want to change the section field you can run "e". Then make your changes and the package is accepted.

If you want to reject the package type "m". Then you get a text editor (vim) and can describe the reason there. In case of rejection please use the following template form:

Hi Maintainer, ...

... reason ...
rejected for now.

At the end of the rejection message lisa will automatically insert the text from the template which can be found under "/org/ftp.skolelinux.org/katie/templates/katie.rejected" Please do not edit the template.

If you accidently rejected a package, you can just move all files (except *.reason) from ./reject to ./unchecked.

Moving packages to other distributions

Packages are usualy requested to be moved into the stable (non -test) by a developer. You can use the status pages for Etch and Lenny and compare with the dynamicaly generated page at http://ftp.skolelinux.org/skolelinux/etch_needs_love.html

Things to look at can be

  • Read the changelog about the changes from the stable package to the proposed package
  • debdiff stablepackage.deb proposedpackage.deb | diffstat
  • debdiff stablepackage.deb proposedpackage.deb | less
  • do NOT move a package, that you yourself have changes in, since last version in stable package.

Packages should be uploaded to lenny-test for now. After they have been there for a while, and the maintainer and/or ftpmaster deceided to move it to lenny, the ftpmasters need to move them by hand. This is to make sure only stable and good packages end up in the stable repository. Packages are moved from sarge-test to sarge, etch-test to etch or from lenny-test to lenny. The command to be executed is as follows

/org/ftp.skolelinux.org/bin/package-sync -s lenny-test package-name

This command will move package-name from etch-test to etch.

/!\ Currently etch is untouchable (=frozen) and this command (alone) doesnt work. See below.

Don't wory about packages that are uploaded with distribution etch in their changelog, they will end up in etch-test first. :-)

package-name is the name of the source package. package-sync then moves all binary packages built from this source package.

when $suite is untouchable

  1. prepare the announcement mail in svn under stable-updates/package.codename.version
  2. if HOME is set incorrectly: export HOME=/var/lib/dak ; export LANG=C

  3. deactivate the cronjobs, vi /etc/cron.d/dak

  4. then copy the dak.conf file away: cp /etc/dak/dak.conf /etc/dak/dak.conf.bak

  5. then remove the "Untouchable" line for the etch suite in katie.conf, vi /etc/dak/dak.conf

  6. then move the package as usual (after checking, etc.)
  7. then run dinstall: /opt/dak/cron/cron.dinstall

  8. then copy the old dak.conf back (with the untouchable line), mv /etc/dak/dak.conf.bak /etc/dak/dak.conf

  9. check the archive by running dak ls $package

  10. first do touch /org/ftp.skolelinux.org/RUN-DINSTALL and run dinstall again /opt/dak/cron/cron.dinstall

  11. and activate the cronjobs again, vi /etc/cron.d/dak

  12. and then check again with dak ls and maybe run dinstall a third time
  13. send the announce mail, gpg signed, to debian-edu-announce@lists.debian.org

  14. update the stable status page, by moving the package from proposed updates to, updated packages on the Stable status page

when $package is out of sync on different archs

_Sometimes_ its useful to migrate a package on one arch only, because the others have not been build yet. In general this should be avoided, but if you have to do it, here is how to clean up the mess:

/!\ As with everything here: only run the commands if you understand what they do!

 dak ls debian-edu
 dak ls education-standalone
 dak ls -a powerpc -s lenny-test debian-edu -S -f heidi
 dak ls -a powerpc -s lenny-test debian-edu -S -f heidi |  dak control-suite -a lenny
 dak ls -a amd64 -s lenny-test debian-edu -S -f heidi
 dak ls -a amd64 -s lenny-test debian-edu -S -f heidi |  dak control-suite -a lenny
 dak ls debian-edu
 dak ls education-standalone

removing packages

to remove packages from the archive: dak rm -s $suite -C debian-edu@lists.debian.org $sourcepackage

about the other options: dak rm -h :)

to remove packages from queue/unchecked (for example, because they were rejected, because the key is not in the keyring): just rm them!

If you want/need to use rene, dont use rene in path but the one in /org/ftp.skolelinux.org/katie/bin/ /!\ FIXME: path does not exist.

Steffen also told us on IRC that we should remove the appropriate .changes file from the done-directory, then run a dinstall and then probably remove the files manually after removing a package with the dak commands. And in the end we of course should check everything :)

If you need to do a complete removal, http://folk.uio.no/werner/ftpmaster_remove_slbackup-php.html has a transcript of a complete removal session.

un-removing packages

If you removed a package with "dak rm" you can un-remove it like this:

echo flashplugin-nonfree-extrasound 0.0.svn2431-2.0.edu.etch.2 source | dak control-suite -a etch-test
echo flashplugin-nonfree-extrasound 0.0.svn2431-2.0.edu.etch.2 i386 | dak control-suite -a etch-test

You need to specify each arch and suite.

New Accounts:

Please note that only drift@skolelinux.org decides to give someone account rights! New contributors should read this first. If drift writes a mail to ftpmasters and tells them that a new Uploader needs an account for upload rights the ftpmaster need the gpg-key from the new Uploader. For that purpose please write the new Uploader direclty and ask after a mail or get the key id and fingerprint from drift@skolelinux.org.

If the ftpmasters are instructed to create an account they can follow this procedure:

After login on administrator.skolelinux.no they have to become katie user ("sudo -u katie bash"). After that they should set the environment variable HOME by typing "export HOME=/var/lib/dak" . Then they can use the command: /org/ftp.skolelinux.org/bin/new-account -k ''keyid''

Now you have to whitelist the uploaders email-address: checkout dak-config via cvs from /org/ftp.skolelinux.org/cvsroot and edit mail-whitelist and commit it with "cvs ci mail-whitelist". Now cd to /org/ftp.skolelinux.org/katie/ and do "cvs up". Please make sure that you use an up to date cvs working copy before editing the file, to prevent conflicts at commit.

After that the new account is ready to be used and the new Uploader will get a mail which informs about it and explains some configuration steps for dput/dupload. Please add the name of the Uploader to the Uploaders-list which can be found under: http://wiki.debian.org/DebianEdu/Uploaders

Adding new keys to old accounts

as user kathie run:

export HOME=/var/lib/dak ; export LANG=C
gpg --keyserver pgpkeys.pca.dfn.de --no-default-keyring --keyring /opt/dak/keyrings/keyring.gpg --recv-keys E63CD6D6  

to add new subkeys for the existing key E63CD6D& in the keyring.

What does each script do

Scripts

  • Katie is just de username under which these scripts run, also see /org/ftp.sk.../cron/
  • melanie: removes packages
  • dak ls - shows which suites/archs a package is in
  • dak control-suite - list packages in suite and moves/copies packages between suites
  • new-account Specific to Debian-edu
    • adds user to keyring and thus gives upload rights for edu-DDs
      • Replaces Emilie in this setup Restricted access (peter and bana) Usage:

        set the $HOME variable to /var/lib/dak, then sabina -k keyid -> sends email with info on upload rights and conf

  • Jennifer run by /cron/unchecked, does the checks for ../unchecked
  • Helena reports by email what's in new or helena -n  gives html output of new (similar to the NEW normal debian thingy) - helena is also run by katies cronjobs

  • lisa is doing new processing and needs to be run manually (after helena mails)
    • (cd $new ; lisa *.changes)
  • kelly - installs packages into the pool
  • jenna - generates lists of files in suites which are then fed to apt-ftparchive
  • ziyi - creates Release files
  • rhona + shania (ignore for now, cleanup stuff)
  • rene: sometimes needs to be run manually to check for obsolete packages (run daily via cron)
  • pgdump (backup the database)
  • Rose - Directory creation
    • reads katie.conf and apt.conf used to add more suits
  • Neve - Database creation
    • only at installation, it drops dtabase (postgresql) super user privileges (from stdin to databases)
  • Billy (not yet needed)
  • Emilie (not used in Debian-edu)
    • GPG signatures checking and put the in the DB (use restricted)
  • Britney is for testing transition in Debian archive (not used in Debian-edu)

(We will upgrade to the new dak version without those scripts after the etch release.)

the projectb database

Although the examples shown here are made as the katie user they should work as well with your user account. But this is only true for these examples, as only katie has write access.

using psql

katie@administrator:~$ psql projectb
Welcome to psql 7.4.19, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help on internal slash commands
       \g or terminate with semicolon to execute query
       \q to quit

projectb=> 

Now you can use normal sql queries to access dak's projectb database.

Entering \z at the psql prompt will list all tables with their access privileges settings, \q quits psql.

This explains how to learn more about the layout of the database:

projectb=> select * from pg_tables where schemaname='public';
 schemaname |      tablename      | tableowner | hasindexes | hasrules | hastriggers 
------------+---------------------+------------+------------+----------+-------------
 public     | architecture        | katie      | t          | f        | t
 public     | archive             | katie      | t          | f        | t
 public     | bin_associations    | katie      | t          | f        | t
 public     | binaries            | katie      | t          | f        | t
 public     | component           | katie      | t          | f        | t
 public     | dsc_files           | katie      | t          | f        | t
 public     | files               | katie      | t          | f        | t
 public     | fingerprint         | katie      | t          | f        | t
 public     | location            | katie      | t          | f        | t
 public     | maintainer          | katie      | t          | f        | t
 public     | override            | katie      | t          | f        | t
 public     | override_type       | katie      | t          | f        | t
 public     | priority            | katie      | t          | f        | t
 public     | section             | katie      | t          | f        | t
 public     | source              | katie      | t          | f        | t
 public     | src_associations    | katie      | t          | f        | t
 public     | suite               | katie      | t          | f        | t
 public     | suite_architectures | katie      | t          | f        | t
 public     | accepted_autobuild  | katie      | f          | f        | t
 public     | uid                 | katie      | t          | f        | t
 public     | queue               | katie      | t          | f        | f
 public     | queue_build         | katie      | f          | f        | f
(22 rows)

projectb=> select * from pg_tables;
projectb=> select * from suite limit 1;
 id | suite_name | version |       origin       | label | policy_engine |    description     
----+------------+---------+--------------------+-------+---------------+--------------------
  5 | lenny      |         | Skolelinux archive |       |               | Skolelinux archive
(1 row)

projectb=> 

using pg_dump

pg_dump projectb conviniently piped into grep allows to easily query the database on the commandline. An example says as much as 1000 words:

katie@administrator:~$ pg_dump projectb | grep multiseat
1825    local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3_amd64.deb     20810   1a6229f78968adfb4d811db7759fc252        2       \N
1826    local/m/multiseat/multiseat-udeb_0.9.10-0.0.edu.etch.3_amd64.udeb       11564   ccb66f75c1afe08d0a9c94080aa9253d        2       \N
699     local/m/multiseat/multiseat_0.9.10.orig.tar.gz  32092   081ff6a0475b1fd44e41975059f383de        2       \N
1476    local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3_powerpc.deb   20868   5abf45a8fddc0238d6919a99fd0664f4        2       \N
1477    local/m/multiseat/multiseat-udeb_0.9.10-0.0.edu.etch.3_powerpc.udeb     11554   cc89256ecb45fd9df8992dad03383c96        2       \N
717     local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3.dsc   613     b5de06c2af35ded51d7d333ec026b398        2       \N
718     local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3.diff.gz       3365    c11aaf4c17903fb79640e9f21cd38f77        2       \N
719     local/m/multiseat/multiseat_0.9.10-0.0.edu.etch.3_i386.deb      20512   77fa29628e752c2dcbbd3ce3d9da45d5        2       \N
720     local/m/multiseat/multiseat-udeb_0.9.10-0.0.edu.etch.3_i386.udeb        11340   bbbabdfe5a3fb8e66420f0e5a18b8ee6        2       \N
118     multiseat       0.9.10-0.0.edu.etch.3   4       717     2006-08-17 00:00:00     2
1255    multiseat       0.9.10-0.0.edu.etch.3   4       118     5       1825    deb     8
1256    multiseat-udeb  0.9.10-0.0.edu.etch.3   4       118     5       1826    udeb    8
466     multiseat       0.9.10-0.0.edu.etch.3   4       118     3       719     deb     2
467     multiseat-udeb  0.9.10-0.0.edu.etch.3   4       118     3       720     udeb    2
984     multiseat       0.9.10-0.0.edu.etch.3   4       118     4       1476    deb     7
985     multiseat-udeb  0.9.10-0.0.edu.etch.3   4       118     4       1477    udeb    7
multiseat       4       2       10      68      4       \N
multiseat-udeb  4       2       9       38      5       \N
katie@administrator:~$ 

dak directory layout

  • /org/ftp.skole..../queue
    • /unchecked -> public incoming dir, cronjob

      • (/cron/unchecked) looks at the changes file -> to new, accepted, rejected...

      /new

      /accepeted -> REPORT -> dinstall report /rejected /done -> after dinstall runs, changes files are here, the ret of the package goes to the pool

      -> REPORT -> file with logs on what happened

Setup

dinstall is run every 15min., looks for changes for in /accepted or (the manually to be created file /org/ftp.skolelinux.org/RUN-DINSTALL), then it creates a lockfile

adding a new suite

  • edit /etc/dak/dak.conf as appropriate
  • edit /etc/dak/apt.conf as appropriate
  • run

touch /org/ftp.skolelinux.org/scripts/override/override.squeeze.local.debian-installer
touch /org/ftp.skolelinux.org/scripts/override/override.squeeze.local
mkdir -p /org/ftp.skolelinux.org/ftp/dists/squeeze-test/local
cd /org/ftp.skolelinux.org/ftp/dists/squeeze-test/local
mkdir binary-amd64  binary-i386  binary-powerpc  debian-installer  source
cd debian-installer/
mkdir binary-amd64  binary-i386  binary-powerpc
  • run psql projectb then

projectb=> INSERT INTO suite VALUES (DEFAULT,'squeeze-test','','Skolelinux archive','','','Skolelinux archive');

adding a new section

  • edit /etc/dak/dak.conf as appropriate
  • run psql projectb then

projectb=> INSERT INTO suite VALUES (DEFAULT,'local/debug');

you can't touch etch

After a release, suites are marked as "untouchable". In order to mark a suite untouchable, one of the ftpmasters adds the line "Untouchable "1";" to the end of the suite configuration in /org/ftp.skolelinux.org/katie/katie.conf . This tells dinstall to avoid touching the suite. If there is a need for a newer package, just uncomment the line from katie.conf and let the dinstall run.

TODO-List for ftpmasters:

  1. there are no mails send for uploads to lenny*
  2. Finish documenting the new dak commands in the ftpmaster-howto.
  3. Put some of the dak stuff under a VCS.
  4. Contact the buildd admins to add the new suites to their system and check the scripts, which are run after dinstall (they generate the package overview page and trigger wanna-build)
  5. needs_love should be installed as katie, not holger
  6. needs_love now compares our packages against those on security.debian.org, should it send a mail in case something is newer there?
  7. run make_summary from a svn checkout
  8. we need to implement the changes into package-sync so that also a specific package version can be choosen
  9. document execution-stay-off-time
  10. document installation/configuration on a.s.n
  11. pere asks: why is the email about the etch update listing one of the binary packages and not the source package? For example education-astronomy - can we change this with reasonable effort?
  12. upgrade to latest dak after etch release
    • we manually fixed /usr/bin/melanie in line 165 (python syntax bug)
  13. sign the md5sum files of the cd/dvd images