Out of Date : The svnbuildstat service described here is currently unavailable. The following is for reference/historical interest only.
Prerequisites: To get the most from this article, understand these concepts and terms before reading: basic Core Utilities, configuration files, INSERT YOUR PREREQUISITES HERE
Table of Contents
It would be really helpful if there would be a tool that does automatic building of Debian packages that are maintained either in Subversion, git, or just uploaded as a source package. The service should provide a Debian repository of these binary packages, all automatically built using a pbuilder, together with lintian and piuparts logs. Anyone registered could upload his source package (maybe only if he provides a build bot first). Just like Ubuntu PPA but for Debian and more integrated with Debian (alioth, BTS, svn.debian.org, git.debian.org, etc.).
This would be helpful for both team and individual maintainance and simply make it easier for new Debian maintainers to create their first packages and make sure they are high quality and also to provide a means to anyone to just fix a package, upload it to svnbuildstat and allow the problem to be fixed not only to him, but to anyone adding the svnbuildstat package repository to his sources.list. The svnbuildstat service should also create a simple patch against the official package in unstable, so that the one, who fixes the problem is encouraged to send the patch to the respective maintainer.
This project has arised from the need to have some automated checks done on the Debian Games Team's packages which are team maintained in a Subversion repository.
Any reason this is being tied just to svn and git? You can easily add arch grab files to this (try baz get http://arch.debian.org/arch/private/srivasta/grab/fvwm). A better name would be VCSBbuildstat, to emphasize this is not just a subversion thing (I almost did not read thie page because of that; I don't do subversion). ManojSrivastava
- October 2006: An initial implementation was done by Gonéri Le Bouder for the Debian Games Team. It had many hard-coded paths and it was strongly tight with the Debian Games Team's development process.
- March 2007: Eddy Petrişor has done some work to remove the hardcoded paths, make the thing more configurable, added support for svn:externals, lots of debranding, fixed some assumptions that worked out of luck
- April 2007: Goneri started to work on svnbuildstat, a complete rewrite which should use a database for tracking information - this code is still under development
1st of May 2007: A first public version of the build results of the new Autobuilder are available at http://nana.rulezlan.org/svnbuildstat/ .
September 2007: new homepage: http://svnbuildstat.debian.net
- can autobuild packages directly from SVN (if a few preconditions are true: access to the SVN repo; access to orig tarballs, if needed)
- generates per maintainer and per team web pages with the packages of each individual is maintainer and/or co-maintainer
- automatic lintian checks
- visually represent with green/yellow/red background build success/ongoing/failed status on the status pages (there is one for each developer and one for the whole team)
- directly link to the BTS bugs of every package, so the status page becomes the central point of coordination for the developer.
- supplemental checks can be added (e.g.: gnome policies)
- custom builder commands can be used, but svn-buildpackage+pbuilder is recommmended; no other builders were tested, but it should be possible to modify the tool to make it possible
- supports building a collection packages located in the same directory or packages gathered in the same directory via svn:externals (this is useful if one uses layout 1 of svn-buildpackage but wants to be able to autobuild and check all trunks)
The source is available in the Debian QA Subversion repository. It can be checked out
svn co svn://svn.debian.org/svn/collab-qa/svnbuildstat
Svnbp uses these scripts to feed/refresh the database:
- svnbuildstat_update-db.pl: read the svn repositories to find which updated package
svnbuildstat_update-bugs.pl: download a dump of the bts2ldap gateway ( http://qa.debian.org/data/bts2ldap/fullindex ) and feed the database this the bugs status
- svnbuildstat_update-repository.pl: refresh the local repository
- svnbuildstat_import-new-report.pl: scan the incoming directory for new report and import them in the DB
- svnbuildstat_purge-buildinprogress.pl: deprecated script
*/30 * * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_update-db.pl > /dev/null 2>&1 5 * * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_update-bugs.pl > /dev/null 2>&1 * * * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_update-repository.pl > /dev/null 2>&1 * * * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_import-new-report.pl > /dev/null 2>&1 0 0 * * * root /home/sites/svnbuildstat.debian.net/svnbuildstat/script/svnbuildstat_purge-buildinprogress.pl > /dev/null 2>&1
TODO list from the next release
- support other vcs: git, arch, bzr, ... (WORK IN PROGRESS)
- svnbuildstat_update-db.pl is subversion specific and need to be splited (DONE)
- svnbuildstat_update-repository.pl is subversion specific (DONE)
- patch for pbuilder (create, send and let accepted): kill the build if there is less than 300Mb free or 100Mb swap memory free
- being able to have more than one repository by team (DONE)
- server and the client shouldn't share the same config file
find a new name since Svnbuildstat will not just be Svn ?QaBuildStat?
- support svn-buildpackage origUrl propert to get the tarball
- support debian/rules targets to prepare a fresh tarball
- the server should be light, i.e. it should only provide the infrastructure, all hard work should be done by "bots":
- buildbot - will download the *.dsc off the web from the server, build the binary packages for some architecture and upload them back to server
- checkbot - will download the source+binary package, check it with lintian and piuparts.
- the bots should be extremely simple to setup, there should be a Debian package for them, the user will just apt-get install a package, and that's it. Ideally no configuration, or very simple. The interface between the server and the bots should be also very simple, so that it's easy to create a new bot, or just download and build the package manually, if something goes wrong.
- Gonéri Le Bouder
- Eddy Petrisor, patches for the autobuild and initial wiki documentation
How to get it working
Install prerequisites (Etch):
aptitude install libcatalyst-perl libcatalyst-modules-perl libdbd-pg-perl postgresql-8.1 subversion libfile-touch-perl dpkg-dev
Install svnbuildstat: Today some paths are still hardcoded and we strongly advise you to use the /home/sites/svnbuildstat.debian.net directory as root directory:
sudo mkdir -p /home/sites/svnbuildstat.debian.net sudo chown username:usergroup /home/sites/svnbuildstat.debian.net cd /home/sites/svnbuildstat.debian.net svn co svn://svn.debian.org/svn/collab-qa/svnbuildstat/branches/stable svnbuildstat cd svnbuildstat vim lib/SvnBuildStat/Config.pm
If you don't want to use the standard path for the config file (/etc/svnbuildstat.conf), edit the lines 7, 8:
$file = "./svnbuildstat.conf";
create the database (enter the password you put in ./svnbuildstat.conf):
$ sudo su postgres $ createuser -P svnbuildstat Enter password for new role: Enter it again: Shall the new role be a superuser? (y/n) n Shall the new role be allowed to create databases? (y/n) n Shall the new role be allowed to create more new roles? (y/n) n CREATE ROLE $ createdb -O svnbuildstat svnbuildstat -E UTF8 CREATE DATABASE $ psql svnbuildstat svnbuildstat=# \i schema-pg.sql
If you prefere to have an initialised database, complet dump can be downloaded here: http://nana.rulezlan.org/~goneri/svnbuildstat/
You also need to setup the work directory:
svnbuildstat_update-repository.pl and svnbuildstat_import-new-report.pl need both to have work directory svnbuildstat_import-new-report.pl use the setting from the [report] section whereas svnbuildstat_import-new-report.pl work directory is hardcoded at the begining of the script (be careful, it plays with rm -Rf) svnbuildstat_update-repository.pl is the last script I did and I still don't have clean up it It will use the svnbuildstat.conf too you'll probably find some new missing dependency with these scripts
run the svnbuildstat:
$ script/svnbuildstat_www_server.pl DBI:Pg:database=svnbuildstat;host=127.0.0.1 svnbuildstat root DBIx::Class::Schema::Loader::connection(): viewchangelog has no primary key at /usr/share/perl5/Catalyst/Model/DBIC/Schema.pm line 292 DBIx::Class::Schema::Loader::connection(): viewpackage has no primary key at /usr/share/perl5/Catalyst/Model/DBIC/Schema.pm line 292 DBIx::Class::Schema::Loader::connection(): viewtobuild has no primary key at /usr/share/perl5/Catalyst/Model/DBIC/Schema.pm line 292 DBIx::Class::Schema::Loader::connection(): viewtobuild2 has no primary key at /usr/share/perl5/Catalyst/Model/DBIC/Schema.pm line 292 You can connect to your server at http://fuji:3000
Point your browser to http://localhost:3000 and enjoy.
aptitude install libwww-perl
To download development version, use:
svn co svn://svn.debian.org/svn/collab-qa/svnbuildstat/trunk svnbuildstat
Development debian repository:
deb http://debppa.certik.cz/debian/ unstable main contrib non-free deb-src http://debppa.certik.cz/debian/ unstable main contrib non-free
aptitude install libcatalyst-perl libcatalyst-modules-perl libdbd-pg-perl postgresql-8.3 subversion libfile-touch-perl dpkg-dev libconfig-inifiles-perl devscripts fakeroot liblogger-syslog-perl (not in Etch but the one from Lenny works fine after a rebuild)
The current devel schema
To refresh it:
postgresql_autodoc -d svnbuildstat -t dot dot -Tpng svnbuildstat.dot > svnbuildstat_devel_db_schema.png
The build bot send a ini file to the server through a HTTP POST. The contains the arch of the buildbot. The server reply by sending the same ini file but with a build id and the URL to the dsc file to buid. Once the build is finish the buildbot send back the ini file and the build result. Take a look on getBuildJob() and postFile() in svnbuildstat_agent.pl.