Package Repositories for Debian Developers

Student Name

Raphael Geissert

Contact/Email:

Abstract

For testing and bug-fixing purposes it is often necessary to modify Debian packages and build them on the supported architectures. Requesting users to build a package by hand makes it an unnecessarily complicated process. By providing a service similar to ppas to Debian Developers and possibly Debian Maintainers, it should be possible to ease testing and development of unofficial packages (but that may exist officially.)

Key features

Content

Based on the idea by Marc Brockschmidt, the current infrastructure used to build the official packages (the wanna-build/buildd-network) could be extended and used to provide this new service. Extra features such as building on a specific architecture, binNMUing a given set of packages, and more could be added. The plan would therefore be to develop and deploy the service as described by Marc Brockschmidt's project idea, with the possibility of changing the implementation details and adding more features that are found useful. Some implementation details such as the archive software that would be used to handle the different repositories need to be evaluated to offer the best service while still being easy to administrate and extend. Being familiar with most parts of the involved infrastructure and even having developed 1 a tool that to some degree is a small-scale version of the wanna-build/buildd network makes me a good candidate that is able to accomplish this project. All this work was done even before I became a Debian Developer.

Other information

Work plan

Although not exhaustive and not yet a defined timeline, this is the initial work plan:

  1. Become familiar with wanna-build and the rest of the buildd infrastructure.
  2. Setup test environment.
  3. Start evaluating the different existing archive management programs.
    1. Requirements: package pools, gpg-based package authentication, multi-repository support, custom keyring between different repositories, upload queue management, wanna-build interface, etc.
  4. Talk with DSA team about possible authentication methods for the web interface.
  5. Add support for different repositories with different sources.list and upload queues requirements. Will require some sort of capabilities check code on the buildds to make sure they can create snapshots so that changes to apt's configuration is only temporary (one-time setups.)
  6. Design requests database. Changes to the repositories (included their very own creation) would be scheduled via a database and a backend would process them asynchronously.
  7. Select archive management software and set it up on the test environment.
  8. Add support for package auto-signing to sbuild and its corresponding key management.
  9. Start discussion with the rest of the developers (via debian-devel@l.d.o) about who should be allowed to upload by other DDs (DMs only? maybe non-DD/DMs package maintainers?) and what (main only? contrib too?), etc.

  10. Initial web interface design
  11. Per-repository user uploads and buildd uploads key management system.
  12. <to be continued>

  13. Setup archive software
  14. Coordination with DSA team to possibly directly implement the system on a debian.org machine.
  15. Setup on a "public" system (d.o machine or other public service).
  16. Test period to a small group of people.
    1. Initially to DDs only.
    2. Later to DDs and DMs.
  17. Open the service to everyone

Proposal comments

Marc: > Which features do you think are hard to implement, which would be easy?

Choosing and setting up the archive management software is probably going to be the hardest part. For example, dak is full of tools but it might be too big for the requirements. Reprepro has many features too but lacks the upload queue management part (which is basically what ?DeBaBaReTools was about). Finally, there are other solutions (launchpad's included) which also need to be evaluated.

The web interface should not be much problem. By properly designing the system, the web interface should do nothing on its own -- the grunt work should be done by the backend. This would allow the interface to be written in PHP or Perl without fears of the code becoming a maintenance burden at a later time.

The authentication system for the web interface can of course not be taken lightly. At present, almost all the different debian.org web services that require some sort of authentication use a different system and usually different credentials that are managed independently from the rest of the services. This part of the project requires discussion, coordination and cooperation with the DSA team to define the best authentication method that suits both parties.

Marc: > What are your plans for the summer? How much time will you have to work on this?

My school term ends by the last week of June. Before it ends I won't be able to spend many hours a day, but I should have enough time to start working on the project. During weekends and after the school term finishes I should be able to dedicate over eight hours a day, probably even more.

Marc: > Are you familar with the current code base of wanna-build and/or buildd? With dak or some other archive tool?

I'm not familiar with the code base of wanna-build or buildd. I've taken a look at some of dak's source code files but not enough to say "I'm familiar" with it. I have not seen reprepro's code either but I have used it in multiple occasions. In fact, the "?DeBaBaReTools" project uses it.

Marc: > Especially the last part of my proposal, concerned with the actual interface to developers who want to set up a repository, is very vague. Could you describe how this could be implemented and how classical problems of user repositories (signing keys, lots of old and non-updated repositories, packages in need of a binNMU for a transition, ...) can be mitigated?

The interface would allow:

The web interface (and of any other kind) would communicate with the backend by the means of a restricted database connection. Actions would be placed by the interfaces into the database, for later processing by the backend. This design permits the distribution of the backend in one or multiple different servers (one backend per server.)

Common problems with repositories and proposed solutions:

* The need of binNMUs: binNMUs should be requested by the repository maintainers, giving them more control over what their packages want to be built against. To ease the maintenance of the repositories, a report based on EDOS-debcheck could be provided, indicating installability issues of the packages in the repository.

Additional info

http://wiki.debian.org/SummerOfCode2010/DeveloperPackageRepositories