Extending Debian repos

We want (761348) to enable:

• Automatic enumeration of suites for a repository
• Automatic enumeration of repositories for a distro

This page is for exploring the existing solutions, existing hardcoding and requirements for services.

Existing solutions

distro-info

Has the same problems as hardcoding elsewhere but better than hardcoding in lots of places.

Ubuntu

http://changelogs.ubuntu.com/meta-release

Tanglu

https://lists.debian.org/CAKNHny8OUnOveJxOmDNhQHcoFJG4PbKgCh4ZBYwuj5QyvF-eFQ@mail.gmail.com

reprepro

Has a configuration file listing all suites with metadata about all of them.

Existing hardcoding

non-free-firmware

This mail details some places that are likely to need to hard-code the new non-free-firmware component.

distro-info

All the info.

sources.debian.org

Name of all suites in the archive, with explicit mappings from "symbolic" names (stable, testing, ...) to "absolute" names (wheezy, jessie, ...). Bonus point: metadata about both current and past releases (on archive.d.o), such as when the releases happened, their version numbers, etc.

Ideally, the above info should be available for all archives, and in particular also for the security archive. Bonus point: having a single central place that gather together information from all relevant archives (the main one, security, what else?).

piuparts.debian.org

Encodes codename combinations like squeeze2bpo2wheezy in a configuration file.

dsa-nagios

Has hostgroups for each Debian release that is currently in use.

distrobadges

Hardcodes latest Ubuntu release, Debian suite names, compression to suite mappings, currently working mirror.

dsa-puppet

www.d.o apache2 config hard-codes codename/version mapping and arch association mapping.

security-tracker

• data/config.json: hard-codes codenames, associated pockets, architectures and mappings to suite names.
• lib/python/debian_support.py: listReleases hardcodes the experimental suite

webmaster-team/cron

Has architectures, sections hard-coded.

• parts/7release-notes hard-codes which codenames to build release notes for and which codename to use for trunk.

• lessoften-parts/1installation-guide hard-codes which suites have the installation-guide

bugs.debian.org

@gTags in /srv/bugs.debian.org/etc/config on bugs-master.debian.org has a list of releases and their -ignore tags.

/srv/bugs.debian.org/source/bugscan/bugcfg.pm has hardcoded list of releases and architectures (used for the britney counts)

webwml

english/releases/*/release.data has architectures, install manual languages and release notes languages.

english/template/debian/installer.wml has which suites have CD images for which arches.

english/releases/Makefile and english/Bugs/pkgreport-opts.inc have lists of releases.

english/security/oval/Makefile and english/security/oval/generate.py have lists of releases and their versions (without minor release number).

*/ports/index.wml has which architectures are official and which are not, would be better to use the data from the archive re officialness.

*/distrib/archive.wml has which releases are on archive.debian.org, would be better to automatically update that info.

wiki.debian.org

DebianDesktop/Artwork DebianArt/Themes DebianReleases

tracker.debian.org

Hardcodes various info about the releases. Hardcodes list of Linux architectures from dpkg.

packages.debian.org

Hardcoded release name, symbolic name references, architectures, and sorting of releases. Also information about what a partial release (like -updates and -security) does sort-of overlay.

qa.debian.org

data/RELEASES
data/ftp/calculate-override-disparities
data/ftp/extract-section-priority
data/debcheck/debcheck
data/qadb/sql/qa_apt_distribution.sql
data/qadb/sql/qa_apt_suite.sql

The qadb apt database hard-codes which suites are actively updated and an architecture list.

buildd.debian.org

valid_suites, valid_arches, ignore_arches & ignore_suites in /srv/buildd.debian.org/etc/buildlogs.conf on the buildd host hard-code various aspects of the archive.

dput-ng

Hardcodes the names of backports suites.

deriv.debian.net

Hardcodes the list of all official and unofficial architecture Debian names.

udd

• Hardcodes Ubuntu release names.
• Hardcodes architecture list of several distribution (including Debian)
• Hardcodes architecture list of Debian testing (udd/bugs_gatherer.pl)
• releases table has hard-coded info about releases
• bts_tags table has hard-coded list of bts tags
• udd has a copy of the BTS config, which contains hardcoded info about release (see above for bugs.debian.org info).
  • udd uses the version in /srv/udd.debian.org/udd/bugs-config, which comes from udd git
  • the version from the BTS is available in /srv/bugs.debian.org/etc/config

Section descriptions

Various packages hard-code section descriptions.

devscripts

chdist hardcodes architecture lists. dch/debchange hardcodes suite names, especially for --bpo and --stable. bts hardcodes the BTS suite/suite-ignore usertags.

developers-reference

Hardcodes suite to version, codename and architecture list mappings.

reportbug

Hardcodes suite to codename mappings, -pu codenames, codenames that binNMUs can be done in.

ftpsync

Hardcodes list of architectures that have ever been on ftp.d.o.

nm.debian.org

nm-templates hardcodes recent release codenames in nm_emeritus.txt

cdimage.debian.org

Hardcodes the codename of Debian testing on the d-i daily builds directories like this:

http://cdimage.debian.org/cdimage/daily-builds/sid_d-i/20160103-4/amd64/

The base for this is in

https://anonscm.debian.org/cgit/debian-cd/setup.git

and there are multiple places in here which hard-code release name, architecture names etc.

debian-cd

tools/start_new_disc stores architecture abbreviations for long architecture names.

debian-cd.debian.net

Hard-codes the CD image links and release versions on the front page.

debian-security-support

The debian-security-support package lists which packages are unsupported for security fixes in various versions of Debian. It hardcodes version numbers (e.g. 7 to 9) of supported releases.

Ideally, this package wouldn't exist, and this functionality would be upstreamed in the repository itself.

manpages.debian.org

Hard-codes the list of released versions (see cmd/debiman/rendermanpage.go lines 27-49) and the current stable version (see internal/redirect/redirect.go line 38).

wanna-build

Hard-codes codenames in bin/do_stats triggers/common triggers/trigger.debian

backports.debian.org

Needs to be updated after a stable release.

Debian IRC factoids

Has a procedure for updating between releases.

Popularity contest

• examples/bin/popcon.pl: hardcodes popularity-contest version numbers for each release codename
• examples/bin/popanal{,3}.py: hardcode popularity-contest version number for stable release suite
• examples/bin/{popcon.pl,popanal{,3}.py}: hardcode the suite names stable testing unstable

debomatic

Hardcodes suite to codename mappings. Example pull request to update them.

preseed.debian.net

Hard-codes stable codename, primary architecture and list of available codenames.

appstream.debian.org

Hard-codes the codenames and components to compute the data for.

Requirements

piuparts.debian.org

Want squeeze2bpo2wheezy to not be magically deleted just because jessie became stable. When new stable releases are done, new piuparts suites (like wheezy2bpo2jessie) should be created, not old ones reused.

(This is the case already.)

Single source of data

For some purposes it would be interesting to have a single source of all metadata (on a per-repo basis and also on a global distro-wide basis) so that full apt metadata doesn't need to be downloaded if it isn't needed. This could also be easier to parse in some cases.

Know which suites are partial

Knowing what suites are "partial" (i.e. don't have a full set of packages) and what parent suite they are associated with (to get the remaining packages) is important for example to be able to do proper dependency analysis on a partial suite by trying to look up missing packages in the parent suite.

Know the order of suites

packages.d.o, sources.d.n, manpages.d.o etc need to know what order to display suites in.

Know the activity status of suites

sources.d.n etc need to know if suites will ever be updated again.

Possible solutions

Some sketches of possible solutions...

Extend deb822 sources.list format

Use an extension of the deb822 sources.list format to describe all Debian URLs, suites, components etc. Place at a canonical location for Debian, an aggregation of all the other ones, sign with a long-lived key. Place at the root of each repository (or in the indicies directory) a file called Sources that lists the sources and their relationships, sign with the current repository key.

Types: deb deb-src
URIs: http://security.debian.org/debian-security/
Suites: testing-security
Components: main contrib non-free
Architectures: amd64 ...

Types: deb deb-src
URIs: http://deb.debian.org/debian/
Suites: testing testing-proposed-updates unstable experimental
Components: main contrib non-free
Architectures: amd64 ...

Types: deb deb-src
URIs: http://deb.debian.org/debian/
Suites: experimental
Components: main contrib non-free
Parent: unstable
Architectures: amd64 ...

Types: deb deb-src
URIs: http://incoming.debian.org/debian-buildd/
Suites: buildd-oldstable-proposed-updates buildd-testing-proposed-updates buildd-unstable buildd-experimental
Components: main contrib non-free
Architectures: amd64 ...

Types: deb deb-src
URIs: http://debug.mirrors.debian.org/debian-debug/
Suites: testing-debug unstable-debug experimental-debug
Components: main contrib non-free
Architectures: amd64 ...

Types: deb deb-src
URIs: http://archive.debian.org/debian/
Codenames: bo buzz hamm ...
Components: main contrib non-free
Architectures: amd64 ...

Aggregate and extend partial Release files

Strip hashes from the Release files and publish them at the root of each repository (or in the indicies directory) a file called Releases that lists the releases and their relationships, sign with the current repository key. Then aggregate those files in a central location, adding URI fields referencing the repositories, sign with a long-lived key. Any stanza that is missing Suite/Codename is used to provide defaults for the following stanzas that do.

Origin: Debian
Label: Debian
Changelogs: http://metadata.ftp-master.debian.org/changelogs/@CHANGEPATH@_changelog
Acquire-By-Hash: yes
Components: main contrib non-free
URI: http://deb.debian.org/debian/
Mirrors: https://anonscm.debian.org/viewvc/webwml/webwml/english/mirror/Mirrors.masterlist?view=co
Date: Fri, 16 Dec 2016 20:38:11 UTC
Valid-Until: Fri, 23 Dec 2016 20:38:11 UTC
Architectures: amd64 arm64 armel armhf hurd-i386 i386 kfreebsd-amd64 kfreebsd-i386 mips mips64el mipsel powerpc ppc64el s390x

Suite: unstable
Codename: sid
Description: Debian x.y Unstable - Not Released

Suite: testing
Codename: stretch
Description: Debian x.y Testing distribution - Not Released