Wiki

• FrontPage
• RecentChanges
• FindPage
• HelpContents
• svnbuildstat

Table of Contents

Abstract

It would be really helpful if there would be a tool that does automatic building of Debian packages that are maintained either in Subversion, git, or just uploaded as a source package. The service should provide a Debian repository of these binary packages, all automatically built using a pbuilder, together with lintian and piuparts logs. Anyone registered could upload his source package (maybe only if he provides a build bot first). Just like Ubuntu PPA but for Debian and more integrated with Debian (alioth, BTS, svn.debian.org, git.debian.org, etc.).

This would be helpful for both team and individual maintainance and simply make it easier for new Debian maintainers to create their first packages and make sure they are high quality and also to provide a means to anyone to just fix a package, upload it to svnbuildstat and allow the problem to be fixed not only to him, but to anyone adding the svnbuildstat package repository to his sources.list. The svnbuildstat service should also create a simple patch against the official package in unstable, so that the one, who fixes the problem is encouraged to send the patch to the respective maintainer.

This project has arised from the need to have some automated checks done on the Debian Games Team's packages which are team maintained in a Subversion repository.

• Any reason this is being tied just to svn and git? You can easily add arch grab files to this (try baz get http://arch.debian.org/arch/private/srivasta/grab/fvwm). A better name would be VCSBbuildstat, to emphasize this is not just a subversion thing (I almost did not read thie page because of that; I don't do subversion). ManojSrivastava

History

• October 2006: An initial implementation was done by Gonéri Le Bouder for the Debian Games Team. It had many hard-coded paths and it was strongly tight with the Debian Games Team's development process.
• March 2007: Eddy Petrişor has done some work to remove the hardcoded paths, make the thing more configurable, added support for svn:externals, lots of debranding, fixed some assumptions that worked out of luck
• April 2007: Goneri started to work on svnbuildstat, a complete rewrite which should use a database for tracking information - this code is still under development

• 1st of May 2007: A first public version of the build results of the new Autobuilder are available at http://nana.rulezlan.org/svnbuildstat/ .

• September 2007: new homepage: http://svnbuildstat.debian.net

Features

• can autobuild packages directly from SVN (if a few preconditions are true: access to the SVN repo; access to orig tarballs, if needed)
• generates per maintainer and per team web pages with the packages of each individual is maintainer and/or co-maintainer
• automatic lintian checks
• visually represent with green/yellow/red background build success/ongoing/failed status on the status pages (there is one for each developer and one for the whole team)
• directly link to the BTS bugs of every package, so the status page becomes the central point of coordination for the developer.
• supplemental checks can be added (e.g.: gnome policies)
• custom builder commands can be used, but svn-buildpackage+pbuilder is recommmended; no other builders were tested, but it should be possible to modify the tool to make it possible
• supports building a collection packages located in the same directory or packages gathered in the same directory via svn:externals (this is useful if one uses layout 1 of svn-buildpackage but wants to be able to autobuild and check all trunks)

Source

The source is available in the Debian QA Subversion repository. It can be checked out

svn co svn://svn.debian.org/svn/collab-qa/svnbuildstat

Source structure

Svnbp uses these scripts to feed/refresh the database:

• svnbuildstat_update-db.pl: read the svn repositories to find which updated package

• svnbuildstat_update-bugs.pl: download a dump of the bts2ldap gateway ( http://qa.debian.org/data/bts2ldap/fullindex ) and feed the database this the bugs status

• svnbuildstat_update-repository.pl: refresh the local repository
• svnbuildstat_import-new-report.pl: scan the incoming directory for new report and import them in the DB
• svnbuildstat_purge-buildinprogress.pl: deprecated script

 */30 *    * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_update-db.pl > /dev/null 2>&1
 5 *    * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_update-bugs.pl > /dev/null 2>&1
 * *    * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_update-repository.pl > /dev/null 2>&1
 * *    * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_import-new-report.pl > /dev/null 2>&1
 0 0    * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_purge-buildinprogress.pl > /dev/null 2>&1

TODO list from the next release

• support other vcs: git, arch, bzr, ... (WORK IN PROGRESS)
• svnbuildstat_update-db.pl is subversion specific and need to be splited (DONE)
• svnbuildstat_update-repository.pl is subversion specific (DONE)
• patch for pbuilder (create, send and let accepted): kill the build if there is less than 300Mb free or 100Mb swap memory free
• being able to have more than one repository by team (DONE)
• server and the client shouldn't share the same config file

• find a new name since Svnbuildstat will not just be Svn ?QaBuildStat?

• support svn-buildpackage origUrl propert to get the tarball
• support debian/rules targets to prepare a fresh tarball

Ideas

• the server should be light, i.e. it should only provide the infrastructure, all hard work should be done by "bots":
  • buildbot - will download the *.dsc off the web from the server, build the binary packages for some architecture and upload them back to server
  • checkbot - will download the source+binary package, check it with lintian and piuparts.
• the bots should be extremely simple to setup, there should be a Debian package for them, the user will just apt-get install a package, and that's it. Ideally no configuration, or very simple. The interface between the server and the bots should be also very simple, so that it's easy to create a new bot, or just download and build the package manually, if something goes wrong.

Who

• Gonéri Le Bouder
• Eddy Petrisor, patches for the autobuild and initial wiki documentation

External Links

• SvnBuildstat.d.n

How to get it working

the server

Install prerequisites (Etch):

aptitude install libcatalyst-perl libcatalyst-modules-perl libdbd-pg-perl postgresql-8.1 subversion libfile-touch-perl dpkg-dev

Install svnbuildstat: Today some paths are still hardcoded and we strongly advise you to use the /home/sites/svnbuildstat.debian.net directory as root directory:

sudo mkdir -p /home/sites/svnbuildstat.debian.net
sudo chown username:usergroup /home/sites/svnbuildstat.debian.net
cd /home/sites/svnbuildstat.debian.net
svn co svn://svn.debian.org/svn/collab-qa/svnbuildstat/branches/stable svnbuildstat
cd svnbuildstat
vim lib/SvnBuildStat/Config.pm

If you don't want to use the standard path for the config file (/etc/svnbuildstat.conf), edit the lines 7, 8:

$file = "./svnbuildstat.conf";

create the database (enter the password you put in ./svnbuildstat.conf):

$ sudo su postgres
$ createuser -P svnbuildstat
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
CREATE ROLE
$ createdb -O svnbuildstat svnbuildstat -E UTF8
CREATE DATABASE
$ psql svnbuildstat
svnbuildstat=# \i schema-pg.sql

If you prefere to have an initialised database, complet dump can be downloaded here: http://nana.rulezlan.org/~goneri/svnbuildstat/

You also need to setup the work directory:

svnbuildstat_update-repository.pl and svnbuildstat_import-new-report.pl need both to have work directory
svnbuildstat_import-new-report.pl use the setting from the [report] section
whereas svnbuildstat_import-new-report.pl work directory is hardcoded at the begining of the script (be careful, it plays with rm -Rf)
svnbuildstat_update-repository.pl is the last script I did and I still don't have clean up it
It will use the svnbuildstat.conf too
you'll probably find some new missing dependency with these scripts

run the svnbuildstat:

$ script/svnbuildstat_www_server.pl
DBI:Pg:database=svnbuildstat;host=127.0.0.1
svnbuildstat
root
DBIx::Class::Schema::Loader::connection(): viewchangelog has no primary key at /usr/share/perl5/Catalyst/Model/DBIC/Schema.pm line 292
DBIx::Class::Schema::Loader::connection(): viewpackage has no primary key at /usr/share/perl5/Catalyst/Model/DBIC/Schema.pm line 292
DBIx::Class::Schema::Loader::connection(): viewtobuild has no primary key at /usr/share/perl5/Catalyst/Model/DBIC/Schema.pm line 292
DBIx::Class::Schema::Loader::connection(): viewtobuild2 has no primary key at /usr/share/perl5/Catalyst/Model/DBIC/Schema.pm line 292
You can connect to your server at http://fuji:3000

Point your browser to http://localhost:3000 and enjoy.

the agent

Install prerequisites:

aptitude install libwww-perl

development version

To download development version, use:

svn co svn://svn.debian.org/svn/collab-qa/svnbuildstat/trunk svnbuildstat

Development debian repository:

deb     http://debppa.certik.cz/debian/ unstable main contrib non-free
deb-src http://debppa.certik.cz/debian/ unstable main contrib non-free

New dependencies:

aptitude install libcatalyst-perl libcatalyst-modules-perl libdbd-pg-perl postgresql-8.3 subversion libfile-touch-perl dpkg-dev  libconfig-inifiles-perl devscripts fakeroot
liblogger-syslog-perl (not in Etch but the one from Lenny works fine after a rebuild)

The current devel schema

To refresh it:

postgresql_autodoc -d svnbuildstat -t dot
dot -Tpng svnbuildstat.dot > svnbuildstat_devel_db_schema.png

server/builbot communication

The build bot send a ini file to the server through a HTTP POST. The contains the arch of the buildbot. The server reply by sending the same ini file but with a build id and the URL to the dsc file to buid. Once the build is finish the buildbot send back the ini file and the build result. Take a look on getBuildJob() and postFile() in svnbuildstat_agent.pl.