Package Repositories for Debian Developers
For testing and bug-fixing purposes it is often necessary to modify Debian packages and build them on the supported architectures. Requesting users to build a package by hand makes it an unnecessarily complicated process. By providing a service similar to ppas to Debian Developers and possibly Debian Maintainers, it should be possible to ease testing and development of unofficial packages (but that may exist officially.)
- Easy creation and usage of repositories.
- Usage statistics.
- For unofficial or test/debug packages.
- No mirroring (repositories should be easy to be moved or located in different servers)
Based on the idea by Marc Brockschmidt, the current infrastructure used to build the official packages (the wanna-build/buildd-network) could be extended and used to provide this new service. Extra features such as building on a specific architecture, binNMUing a given set of packages, and more could be added. The plan would therefore be to develop and deploy the service as described by Marc Brockschmidt's project idea, with the possibility of changing the implementation details and adding more features that are found useful. Some implementation details such as the archive software that would be used to handle the different repositories need to be evaluated to offer the best service while still being easy to administrate and extend. Being familiar with most parts of the involved infrastructure and even having developed 1 a tool that to some degree is a small-scale version of the wanna-build/buildd network makes me a good candidate that is able to accomplish this project. All this work was done even before I became a Debian Developer.
- I do intend to attend DC10.
Other summer plans:
- besides DC10, none at the moment.
Exams and other commitments:
- I'm going to have exams during June.
- provide a system to allow the creation of personal, auto-built, repositories.
- auto-built packages auto-signing.
- possibly others.
If you are a Debian Developer: What added value can you bring to this project by doing it as part of the Summer of Code rather than as part of your usual Debian work?
- First of all, because otherwise I probably wouldn't do it. I'm taking GSoC as an opportunity to work on something different than what I usually work on. If my proposal is not accepted I probably would still try to contribute some changes to sbuild to allow auto-signing, which is a feature that many people want and that would take some load off the buildd administrators.
Although not exhaustive and not yet a defined timeline, this is the initial work plan:
- Become familiar with wanna-build and the rest of the buildd infrastructure.
- Setup test environment.
- Start evaluating the different existing archive management programs.
- Requirements: package pools, gpg-based package authentication, multi-repository support, custom keyring between different repositories, upload queue management, wanna-build interface, etc.
- Talk with DSA team about possible authentication methods for the web interface.
- Add support for different repositories with different sources.list and upload queues requirements. Will require some sort of capabilities check code on the buildds to make sure they can create snapshots so that changes to apt's configuration is only temporary (one-time setups.)
- Design requests database. Changes to the repositories (included their very own creation) would be scheduled via a database and a backend would process them asynchronously.
- Select archive management software and set it up on the test environment.
- Add support for package auto-signing to sbuild and its corresponding key management.
Start discussion with the rest of the developers (via firstname.lastname@example.org) about who should be allowed to upload by other DDs (DMs only? maybe non-DD/DMs package maintainers?) and what (main only? contrib too?), etc.
- Initial web interface design
- Per-repository user uploads and buildd uploads key management system.
<to be continued>
- Setup archive software
- Coordination with DSA team to possibly directly implement the system on a debian.org machine.
- Setup on a "public" system (d.o machine or other public service).
- Test period to a small group of people.
- Initially to DDs only.
- Later to DDs and DMs.
- Open the service to everyone
Marc: > Which features do you think are hard to implement, which would be easy?
Choosing and setting up the archive management software is probably going to be the hardest part. For example, dak is full of tools but it might be too big for the requirements. Reprepro has many features too but lacks the upload queue management part (which is basically what ?DeBaBaReTools was about). Finally, there are other solutions (launchpad's included) which also need to be evaluated.
The web interface should not be much problem. By properly designing the system, the web interface should do nothing on its own -- the grunt work should be done by the backend. This would allow the interface to be written in PHP or Perl without fears of the code becoming a maintenance burden at a later time.
The authentication system for the web interface can of course not be taken lightly. At present, almost all the different debian.org web services that require some sort of authentication use a different system and usually different credentials that are managed independently from the rest of the services. This part of the project requires discussion, coordination and cooperation with the DSA team to define the best authentication method that suits both parties.
Marc: > What are your plans for the summer? How much time will you have to work on this?
My school term ends by the last week of June. Before it ends I won't be able to spend many hours a day, but I should have enough time to start working on the project. During weekends and after the school term finishes I should be able to dedicate over eight hours a day, probably even more.
Marc: > Are you familar with the current code base of wanna-build and/or buildd? With dak or some other archive tool?
I'm not familiar with the code base of wanna-build or buildd. I've taken a look at some of dak's source code files but not enough to say "I'm familiar" with it. I have not seen reprepro's code either but I have used it in multiple occasions. In fact, the "?DeBaBaReTools" project uses it.
Marc: > Especially the last part of my proposal, concerned with the actual interface to developers who want to set up a repository, is very vague. Could you describe how this could be implemented and how classical problems of user repositories (signing keys, lots of old and non-updated repositories, packages in need of a binNMU for a transition, ...) can be mitigated?
The interface would allow:
- The creation of repositories
- Including adding/removing administrators and uploaders. The former would be done based on user accounts and the latter by gpg keys (probably taken from the debian-keyring to restrict uploaders to DD/DM, but needs to be discussed).
- Adding/removing repositories it will depend on.
- General repository maintenance: package removal, binNMU/give-back requests.
- Browsing build logs
- Disabling and removing the repository.
The web interface (and of any other kind) would communicate with the backend by the means of a restricted database connection. Actions would be placed by the interfaces into the database, for later processing by the backend. This design permits the distribution of the backend in one or multiple different servers (one backend per server.)
Common problems with repositories and proposed solutions:
- Signing keys: each repository would have its own signing key. Each of these per-repository keys would be generated automatically and signed by another key (the "master repository key," or MRK). The master repository key would be generated and signed by one of the maintainers of this new system (e.g. me and anyone else who wishes to help maintain it).
- Old repositories: considering that the repositories are not mirrored, it should be possible to generate usage statistics from the httpd logs. This data and the date of the last update could be used as metrics to determine when a repository is old. Old repositories would be automatically disabled unless the responsible DD indicates that it should not be disabled by responding to an automatically-generated email. The response to this email should be signed and processed by a backend that would store a given "reactivation period" in a database. Failure to respond to this email in a grace period of time (possibly two months) would lead to the repository getting disabled. Two months later after the deactivation of a repository would lead to its complete removal. The responsible DD would be allowed to reactivate a repository even after it is disabled (but of course before its removal.)
* The need of binNMUs: binNMUs should be requested by the repository maintainers, giving them more control over what their packages want to be built against. To ease the maintenance of the repositories, a report based on EDOS-debcheck could be provided, indicating installability issues of the packages in the repository.