Differences between revisions 1 and 53 (spanning 52 versions)
Revision 1 as of 2014-09-17 08:05:30
Size: 589
Editor: PaulWise
Comment: initial page
Revision 53 as of 2016-08-14 10:27:09
Size: 5793
Editor: PaulWise
Comment: more devscripts hardcoding
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= Extending the Debian repository format to allow suites enumeration = = Extending Debian repos =

We want (DebianBug:761348) to enable:

 * Automatic enumeration of suites for a repository
 * Automatic enumeration of repositories for a distro
Line 4: Line 9:

<<TableOfContents()>>
Line 17: Line 24:
[https://lists.debian.org/CAKNHny8OUnOveJxOmDNhQHcoFJG4PbKgCh4ZBYwuj5QyvF-eFQ@mail.gmail.com] https://lists.debian.org/CAKNHny8OUnOveJxOmDNhQHcoFJG4PbKgCh4ZBYwuj5QyvF-eFQ@mail.gmail.com

=== reprepro ===

Has a configuration file listing all suites with metadata about all of them.
Line 21: Line 32:
=== distro-info ===

All the info.
Line 23: Line 38:
Please fill me in zack :) Name of all suites in the archive, with explicit mappings from "symbolic" names (stable, testing, ...) to "absolute" names (wheezy, jessie, ...). Bonus point: metadata about both current and past releases (on archive.d.o), such as when the releases happened, their version numbers, etc.

Ideally, the above info should be available for all archives, and in particular also for the security archive. Bonus point: having a single central place that gather together information from all relevant archives (the main one, security, what else?).

=== piuparts.debian.org ===

Encodes codename combinations like squeeze2bpo2wheezy in a configuration file.

=== dsa-nagios ===

Has hostgroups for each Debian release that is currently in use.

=== dsa-puppet ===

[[https://anonscm.debian.org/cgit/mirror/dsa-puppet.git/tree/modules/roles/templates/apache-www.debian.org.erb|www.d.o apache2 config]] hard-codes codename/version mapping and arch association mapping.

=== secure-testing ===

 * bin/gen-DSA: hard-codes mappings from oldstable/stable/testing to codenames.
 * security-tracker code: cf. DebianBug:783491

=== debwww/cron ===

Has architectures, sections hard-coded, including non-US.

parts/7release-notes hard-codes which release notes to build and which release to use for trunk.

=== bugs.debian.org ==

@gTags in /srv/bugs.debian.org/etc/config on bugs-master.debian.org has a list of releases and their -ignore tags.

=== webwml ===

english/releases/*/release.data has architectures, install manual languages and release notes languages.

english/releases/Makefile and english/Bugs/pkgreport-opts.inc have lists of releases.

*/ports/index.wml has which architectures are official and which are not, would be better to use the data from the archive re officialness.

*/distrib/archive.wml has which releases are on archive.debian.org, would be better to automatically update that info.

=== tracker.debian.org ===

Hardcodes various info about the releases.
Hardcodes list of Linux architectures from dpkg.

=== packages.debian.org ===

Hardcoded release name, symbolic name references, architectures, and sorting of releases. Also information about what a partial release (like -updates and -security) does sort-of overlay.

=== qa.debian.org ===

{{{
data/RELEASES
data/ftp/calculate-override-disparities
data/ftp/extract-section-priority
data/debcheck/debcheck
wml/debcheck.wml
}}}

The qadb apt database hard-codes which suites are actively updated.

=== dput-ng ===

Hardcodes the names of backports suites.

=== deriv.debian.net ===

Hardcodes the list of all official and unofficial architecture Debian names.

=== udd ===

 * Hardcodes Ubuntu release names.
 * Hardcodes architecture list of several distribution (including Debian)

=== Section descriptions ===

Various packages [[https://lists.debian.org/20150511143710.GA28568@gaara.hadrons.org|hardcode]] section descriptions.

=== devscripts ===

chdist hardcodes architecture lists.
debchange hardcodes suite names.
bts hardcodes the BTS suite/suite-ignore usertags.

=== developers-reference ===

Hardcodes suite to version, codename and architecture list mappings.

=== reportbug ===

Hardcodes suite to codename mappings, -pu codenames, codenames that binNMUs can be done in.

=== ftpsync ===

Hardcodes list of architectures that have ever been on ftp.d.o.

=== cdimage.debian.org ===

Hardcodes the codename of Debian testing on the d-i daily builds directories like this:

http://cdimage.debian.org/cdimage/daily-builds/sid_d-i/20160103-4/amd64/

The base for this is in

https://anonscm.debian.org/cgit/debian-cd/setup.git

and there are multiple places in here which hard-code release name,
architecture names etc.

== Requirements ==

=== piuparts.debian.org ===

Want squeeze2bpo2wheezy to not be magically deleted just because jessie became stable. When new stable releases are done, '''new''' piuparts suites (like wheezy2bpo2jessie) should be created, not old ones reused.

=== Single source of data ===

For some purposes it would be interesting to have a single source of all metadata (on a per-repo basis and also on a global distro-wide basis) so that full apt metadata doesn't need to be downloaded if it isn't needed. This could also be easier to parse in some cases.

=== Know which suites are partial ===

Knowing what suites are "partial" (i.e. don't have a full set of packages) and what parent suite they are associated with (to get the remaining packages) is important for example to be able to do proper dependency analysis
on a partial suite by trying to look up missing packages in the parent
suite.

=== Know the order of suites ===

packages.d.o, sources.d.n etc need to know what order to display suites in.

=== Know the activity status of suites ===

sources.d.n etc need to know if suites will ever be updated again.

Extending Debian repos

We want (761348) to enable:

  • Automatic enumeration of suites for a repository
  • Automatic enumeration of repositories for a distro

This page is for exploring the existing solutions, existing hardcoding and requirements for services.

Existing solutions

distro-info

Has the same problems as hardcoding elsewhere but better than hardcoding in lots of places.

Ubuntu

http://changelogs.ubuntu.com/meta-release

Tanglu

https://lists.debian.org/CAKNHny8OUnOveJxOmDNhQHcoFJG4PbKgCh4ZBYwuj5QyvF-eFQ@mail.gmail.com

reprepro

Has a configuration file listing all suites with metadata about all of them.

Existing hardcoding

distro-info

All the info.

sources.debian.net

Name of all suites in the archive, with explicit mappings from "symbolic" names (stable, testing, ...) to "absolute" names (wheezy, jessie, ...). Bonus point: metadata about both current and past releases (on archive.d.o), such as when the releases happened, their version numbers, etc.

Ideally, the above info should be available for all archives, and in particular also for the security archive. Bonus point: having a single central place that gather together information from all relevant archives (the main one, security, what else?).

piuparts.debian.org

Encodes codename combinations like squeeze2bpo2wheezy in a configuration file.

dsa-nagios

Has hostgroups for each Debian release that is currently in use.

dsa-puppet

www.d.o apache2 config hard-codes codename/version mapping and arch association mapping.

secure-testing

  • bin/gen-DSA: hard-codes mappings from oldstable/stable/testing to codenames.
  • security-tracker code: cf. 783491

debwww/cron

Has architectures, sections hard-coded, including non-US.

parts/7release-notes hard-codes which release notes to build and which release to use for trunk.

=== bugs.debian.org ==

@gTags in /srv/bugs.debian.org/etc/config on bugs-master.debian.org has a list of releases and their -ignore tags.

webwml

english/releases/*/release.data has architectures, install manual languages and release notes languages.

english/releases/Makefile and english/Bugs/pkgreport-opts.inc have lists of releases.

*/ports/index.wml has which architectures are official and which are not, would be better to use the data from the archive re officialness.

*/distrib/archive.wml has which releases are on archive.debian.org, would be better to automatically update that info.

tracker.debian.org

Hardcodes various info about the releases. Hardcodes list of Linux architectures from dpkg.

packages.debian.org

Hardcoded release name, symbolic name references, architectures, and sorting of releases. Also information about what a partial release (like -updates and -security) does sort-of overlay.

qa.debian.org

data/RELEASES
data/ftp/calculate-override-disparities
data/ftp/extract-section-priority
data/debcheck/debcheck
wml/debcheck.wml

The qadb apt database hard-codes which suites are actively updated.

dput-ng

Hardcodes the names of backports suites.

deriv.debian.net

Hardcodes the list of all official and unofficial architecture Debian names.

udd

  • Hardcodes Ubuntu release names.
  • Hardcodes architecture list of several distribution (including Debian)

Section descriptions

Various packages hardcode section descriptions.

devscripts

chdist hardcodes architecture lists. debchange hardcodes suite names. bts hardcodes the BTS suite/suite-ignore usertags.

developers-reference

Hardcodes suite to version, codename and architecture list mappings.

reportbug

Hardcodes suite to codename mappings, -pu codenames, codenames that binNMUs can be done in.

ftpsync

Hardcodes list of architectures that have ever been on ftp.d.o.

cdimage.debian.org

Hardcodes the codename of Debian testing on the d-i daily builds directories like this:

http://cdimage.debian.org/cdimage/daily-builds/sid_d-i/20160103-4/amd64/

The base for this is in

https://anonscm.debian.org/cgit/debian-cd/setup.git

and there are multiple places in here which hard-code release name, architecture names etc.

Requirements

piuparts.debian.org

Want squeeze2bpo2wheezy to not be magically deleted just because jessie became stable. When new stable releases are done, new piuparts suites (like wheezy2bpo2jessie) should be created, not old ones reused.

Single source of data

For some purposes it would be interesting to have a single source of all metadata (on a per-repo basis and also on a global distro-wide basis) so that full apt metadata doesn't need to be downloaded if it isn't needed. This could also be easier to parse in some cases.

Know which suites are partial

Knowing what suites are "partial" (i.e. don't have a full set of packages) and what parent suite they are associated with (to get the remaining packages) is important for example to be able to do proper dependency analysis on a partial suite by trying to look up missing packages in the parent suite.

Know the order of suites

packages.d.o, sources.d.n etc need to know what order to display suites in.

Know the activity status of suites

sources.d.n etc need to know if suites will ever be updated again.