Differences between revisions 1 and 129 (spanning 128 versions)
Revision 1 as of 2005-02-16 20:01:45
Size: 4164
Editor: anonymous
Comment:
Revision 129 as of 2021-01-16 07:10:51
Size: 21364
Editor: PaulWise
Comment: add Pulp
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
## Auto-converted by kwiki2moinmoin v2005-10-07
This documents summarises what I have found about settling a Debian repository.

I have taken care of providing the most correct information at the time of writing but if you find any inaccuracy, please, fix it.

In [http://www.debian.org/doc/manuals/repository-howto/repository-howto.html Aaron Isotton how-to] two kinds of repositories are defined, the trivial one (where the user of the repository has to specify an exact path to the
packages) and the automatic one (that mimic the official Debian
repositories where only the base path, distribution and component have
to be specified and the apt-get magic occurs). His how-to describes
how to setup trivial repositories with dpkg-scansources and
dpkg-scanpackages. Such a topic is also covered by [Oohara Yuuma how-to
http://www.interq.or.jp/libra/oohara/apt-gettable/apt-gettable/]
and [apt how-to
http://www.debian.org/doc/manuals/apt-howto/ch-basico.html#s-dpkg-scanpackages].

The settling of automatic repositories is covered by [http://familiasanchez.net/~sanchezr/?page=debrepository Roberto Sanchez
how-to] who
gives relevant scripts. Pools are nor covered there.

There are also packaged (or not) tools in Debian in order to perform
the task.
## page was renamed from HowToSetupADebianRepository
#language en
~-[[DebianWiki/EditorGuide#translation|Translation(s)]]: [[ko/HowToSetupADebianRepository|한국어(Korean)]]-~
----
This document summarizes the process of setting up a [[DebianRepository|Debian package repository]]. It does not describe the [[DebianRepository/Format|format of a Debian repository]].

{{{#!wiki important
Care has been taken to provide the most accurate information at the time of writing. Please fix any identified mistakes.
}}}

<<TableOfContents>>

== Debian Repository Types ==

There are 2 kinds of repositories from user's perspective:

|| '''archive style''' || '''apt line''' || '''apt-pinning''' || '''secure APT''' || '''status''' ||
|| official archive || "{{{deb http://example.org/debian unstable main}}}" || Yes || Yes || preferred ||
|| trivial archive || "{{{deb http://example.org/debian ./}}}" || No || Yes || deprecated ||

These archives have different meta-data structure. Both archives can store actual package files. Many older repository HOWTOs (e.g. old "Debian Reference (sarge)" and "APT HOWTO (sarge)") address creation of a "trivial archive" and are problematic since the "trivial archive" lacks support for '''apt-pinning''' meta-data used by [[AptPreferences|APT Preferences]] due to the collision of 2 types of Release files.

For the [[SecureApt|secure APT]] compatibility, the modern package archive must be signed by GPG.

References:
 * [[https://www.debian.org/doc/manuals/securing-debian-manual/deb-pack-sign.en.html|Package signing in Debian]] (DDP: Securing Debian Manual)
 * [[https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_debian_package_management_internals|Debian package management internals]] (DDP: Debian Reference)

== Repository instructions ==

The best practices for creating repository usage instructions are available in RepositoryInstructions.

== Debian Repository Generation Tools ==

The full package archive similar to the official archive can be created using:
 * dak
 * mini-dak

The Private Package Archive (PPA) can be created on a web server with a shell account using:
 * reprepro
 * mini-dinstall

people.debian.org is installed with these packages. The PPA archives created on these hosts should only be used for small low-volume experimental archives only.

{{{#!wiki important
Do not run high-volume repositories without consulting the host server's maintainer(s).
}}}
Line 24: Line 50:
 * Goals: Packaging of the tools handling the official Debian repositories.
 * Pro: Real stuff.
 * Cons: Depends on python and ["postgreSQL"] (even if on an other host).
 * Package: [http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=284370 ITP with link to existing package]
 * Distributions: Not yet.
 * Dependencies: python2.3, gnupg, python-ldap, prostgresql
 * Goals: Hosting the official Debian repositories.
 * Pros:
   * It's the official solution
 * Cons:
   * Depends on python and PostgreSQL
   * Lack of documentation
   * Designed for large repositories.
 * Download: [[https://salsa.debian.org/ftp-team/dak|git repository]].
 * Distributions: not in Debian
 * Dependencies:
   * python
   * postgresql (optional)
Line 34: Line 66:

=== debpool ===
 * Goals: Lightweight replacement for dak using a pool layout.
 * Pro: No external dependencies.
 * Cons:
 * Package: [http://packages.debian.org/experimental/devel/debpool Package in experimental]
 * Distributions: experimental
 * Dependencies: perl (optionally gnupg)
 * Automatic repositories: Yes
 * Incoming mechanism: Yes
 * Pools: Yes
 * GPG signing: Yes (with gnupg).

=== reprepro (formerly known as mirrorer) ===
 * Inclusion of .deb without .changes: No
 * Several versions of each package: not in the same Packages instance
 * Wiki Page: [[dak]]

=== mini-dak ===
 * Goals: Hosting new Debian architectures (Partial and lightweight reimplementation of dak in shell script and with no database dependencies)
 * Pros:
   * Easy to setup: edit a config file and run a script to generate the whole structure
   * No database (the pool is the database)
   * All .changes files kept for later possible importing into the master repository
   * Supports mail notifications and does extensive logging
   * Auto package obsoleting
   * Repository snapshotting
   * Supports multiple suites from the Distribution field on the .changes file
   * Additionally supports multipool (splitting each arch into its own pool, to ease partial mirroring)
   * Supports upload ACLs based on gpg public keys
   * Mirror push via ssh
 * Cons:
   * Slow on huge repositories (due to not using a real db mainly)
   * Has been written and tested mainly as a slave archive, so might have some hardcoded stuff which should be fixed to make it work as a master server
   * Still has some quirks to be fixed
 * Download: https://www.hadrons.org/~guillem/debian/mini-dak/
 * Distributions: not in Debian
 * Dependencies: ('grep Requires: *' on the source tree)
   * apt-utils
   * procmail
   * gnupg
   * wget
   * ssh (optional)
   * bzip2 (optional)
   * quinn-diff (optional)
 * Automatic repositories: Yes
 * Incoming mechanism: Yes
 * Pools: Yes
 * GPG signing: Yes
 * Inclusion of .deb without .changes: No
 * Several versions of each package: ??
 * Sites using it:
   * [[http://ftp.ports.debian.org/debian-ports/|unofficial Debian ports]] ([[https://salsa.debian.org/debian-ports-team/mini-dak|mini-dak.git]])

=== reprepro ===
 * (formerly known as mirrorer)
Line 49: Line 109:
 * Pro: Strict checking of what comes in, no database server needed.
 * Cons:
 * Package: [http://alioth.debian.org/project/showfiles.php?group_id=30039 Packages in Alioth repositories]
 * Distributions: Not yet.
 * Dependencies:
 * Pros:
   * Strict checking of what comes in, no database server needed.
 * Cons:
   * Lots of different options and commands, difficult to use for beginners.
 * Package: DebianPts:reprepro
 * Distributions: [[DebianPkg:stable/utils/reprepro|stable]], [[DebianPkg:testing/utils/reprepro|testing]], [[DebianPkg:unstable/utils/reprepro|unstable]] and [[DebianPkg:/squeeze-backports/reprepro|squeeze-backports]]
 * Dependencies: DebianPkg:unstable/utils/reprepro
Line 58: Line 120:
 * Inclusion of .deb without .changes: Yes
 * Several versions of each package: not in the same Packages instance
 * Manual: [[https://salsa.debian.org/brlink/reprepro/-/blob/debian/docs/manual.html|Yes]]
 * HOWTOs:
  * [[https://salsa.debian.org/brlink/reprepro/-/blob/debian/docs/short-howto|Short HOWTO]] (official)
  * [[DebianRepository/SetupWithReprepro|Setting up signed Debian repository with reprepro]]
  * Setting up your own automatic Debian repository see [[https://debian-administration.org/article/286/Setting_up_your_own_APT_repository_with_upload_support|this article on reprepro]]. The link is rather outdated, but still contains some useful information.
  * Setting up reprepro with apache2 and sign key (in italian language) [[https://web.archive.org/web/20190427031534/http://www.fabreg.it/reprepro-repository-debian-43/]]
  * [[https://scotbofh.wordpress.com/2011/04/26/creating-your-own-signed-apt-repository-and-debian-packages/|Creating your own Signed APT Repository and Debian Packages]] by Jon Cowie

=== mini-dinstall ===
 * Goals: Miniature version of dak.
 * Pros:
   * Doesn't require a PostgreSQL database.
   * Small footprint
 * Cons:
   * ?
 * Package: DebianPts:mini-dinstall
 * Distributions: [[DebianPkg:stable/devel/mini-dinstall|stable]], [[DebianPkg:testing/devel/mini-dinstall|testing]], [[DebianPkg:unstable/devel/mini-dinstall|unstable]]
 * Dependencies: DebianPkg:unstable/devel/mini-dinstall
  * apt-utils
  * python
  * python-apt
 * Automatic repositories: Yes (?)
 * Incoming mechanism: Yes
 * Pools: No
 * GPG signing: Yes (external script and setup example provided in documentation)
 * Inclusion of .deb without .changes: No
 * Several versions of each package: Yes (see keep_old option)
 * HOWTO:
  * [[https://upsilon.cc/~zack/blog/posts/2009/04/howto:_uploading_to_people.d.o_using_dput/|dput + mini-dinstall to ease people.d.o uploads]] by Stefano Zacchiroli.
   * This guides you to setup mini-dinstall on [[https://people.debian.org|people.debian.org]]. It supports [[SecureApt|secure apt]] and remote signing.
Line 61: Line 155:
 * Pro:
 * Cons:
 * Package:
 * Distributions: Woody (0.0.18), Sarge (0.1.5), Sid
 * Dependencies: adduser, apt-utils, opalmod (perl modules)
 * Automatic repositories: Yes
 * Incoming mechanism: Yes
 * Pros:
   * Easy to use incoming mechanism - even on remote systems - by using a cron-job
   * Packages can be moved into a distribution by
     1. reading the Distribution value from .changes file or
     1. directly putting the whole package into a distributions-incoming directory.
  * Standard repository (can be pinned)
 * Cons:
   * No Pool-architecture at the moment
   * Some useful checks are missing
   * Cleaning needs to be done manually
 * Package: DebianPts:debarchiver
 * Distributions: [[DebianPkg:oldstable/devel/debarchiver|oldstable]], [[DebianPkg:stable/devel/debarchiver|stable]], [[DebianPkg:testing/devel/debarchiver|testing]], [[DebianPkg:unstable/devel/debarchiver|unstable]]
 * Dependencies: DebianPkg:unstable/devel/debarchiver
  * adduser
  * apt-utils (recommended) | dpkg-dev
  * opalmod (Perl modules)
  * gnupg (optional)
 * Automatic repositories: Yes
 * Incoming mechanism: Yes
 * Pools: No (but suggested somewhere at [[DebianBug:debarchiver|BTS]]).
 * GPG signing: Yes (with gnupg, post-Sarge feature).
 * Inclusion of .deb without .changes: ??
 * Several versions of each package: ??
 * HOWTOs:
  * [[https://web.archive.org/web/20050410000129/http://vipie.studentenweb.org/dev/debarchiver/|debarchiver how-to]].
  * [[https://web.archive.org/web/20060510205006/http://foss.stat.unipd.it/mediawiki/index.php/Debian_Mirror_Setup|Italian howto]] for local Debian package mirroring (similar to apt-proxy).

=== debpool ===

 * Goals: Lightweight replacement for dak using a pool layout.
 * Pros:
   * No external dependencies
   * Easy to use incoming mechanism
   * Standard repository (can be pinned)
 * Cons:
   * Not actively maintained since 2008-10-30 (see [[http://git.debian.org/?p=debpool/debpool.git|latest development]])
   * no checking of older packages being replaced with new ones
   * no notification of what is going on (no mails when new packages are added)
 * Package: DebianPts:debpool (REMOVED)
 * Distributions: [[DebianBug:585390|experimental]]
 * Dependencies:
  * perl
  * gnupg (optional)
 * Automatic repositories: Yes
 * Incoming mechanism: Yes
 * Pools: Yes
 * GPG signing: Yes (with gnupg).
 * Inclusion of .deb without .changes: ??
 * Several versions of each package: ??
 * Wiki page: [[debpool]]

=== DebMarshal ===
 * Goals: Maintain multiple snapshots from upstream distros, to permit staging.
 * Pros:
   * Fast
   * No database server needed (BerkeleyDB)
 * Cons:
   * Lack of documentation
   * Hasn't been released (No version available, SVN repo has only trunk).
 * Download: http://code.google.com/p/debmarshal/
 * Distributions: not in Debian
 * Automatic repositories: Yes
 * Incoming mechanism: Yes
 * Pools: Yes
 * GPG signing: Yes
 * Inclusion of .deb without .changes: ??
 * Several versions of each package: ??
 * Wiki page: [[http://code.google.com/p/debmarshal/w/list|debmartial]]
 * Presentation: [[http://code.google.com/p/debmarshal/downloads/list|ODP and PDF files]]


Built by Google for their use.

=== apt-ftparchive ===
 * Goals: Superset of dpkg-scanpackages and dpkg-scansources.
 * Pros:
   * Does not rely on any external programs aside from gzip.
   * Creates Release and Contents files without providing *.changes
 * Cons:
   * Can be slow on large repositories, unless the input file (FileList) is sorted first (the sort command works).
   * ?
 * Package: DebianPts:apt
 * Distributions: [[DebianPkg:oldstable/admin/apt-utils|oldstable]], [[DebianPkg:stable/admin/apt-utils|stable]], [[DebianPkg:testing/admin/apt-utils|testing]], [[DebianPkg:unstable/admin/apt-utils|unstable]], [[DebianPkg:experimental/admin/apt-utils|experimental]]
 * Dependencies: DebianPkg:unstable/admin/apt-utils
 * Automatic repositories: No (Yes with dupload)
 * Incoming mechanism: No (Yes with custom move cron script with dupload)
 * Pools: Yes
 * GPG signing: No (Yes with dupload with script)
 * Inclusion of .deb without .changes: Yes
 * Several versions of each package: Yes
 * HOWTOs:
  * [[http://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_small_public_package_archive|Debian Reference (lenny)]] -- SecureAPT in combination with dupload (aimed at someone who has shell access to a web server)
  * apt-ftparchive generate [[http://familiasanchez.net/~roberto/howtos/debrepository|Roberto Sanchez how-to]] -- he now recommends to use '''reprepro'''

<<Anchor(trivialArchive)>>
==== Quick instructions to create a trivial local archive with apt-ftparchive ====
We use {{{/var/lib/mydebs}}} as the example location
  * Create the directory {{{
  mkdir -p /var/lib/mydebs
  }}}

  * Every time you put new files in the directory, you'll have to scan the directory and update: {{{
  cd /var/lib/mydebs
  apt-ftparchive packages . > Packages
  apt-ftparchive release . > Release
  }}}

  * Add this line to {{{/etc/apt/sources.list}}} {{{
  deb [allow-insecure=yes] file:/var/lib/mydebs ./
  }}}

  * Now you can install the package normally. If apt-get asks "Install these packages without verification?", answer "Y" to install. That's because this local repository is not signed. {{{
  apt-get update
  apt-get install mydeb
  }}}

=== dpkg-scanpackages and dpkg-scansources ===
 * Goals: Generate Packages index files
 * Pros:
   * Use only bare bone tool provided by dpkg
   * Creates Release and Contents files without providing *.changes with additional scripts.
 * Cons:
   * Cannot create Release or Contents files by themselves
 * Package: DebianPts:dpkg
 * Distributions: [[DebianPkg:oldstable/utils/dpkg-dev|oldstable]], [[DebianPkg:stable/utils/dpkg-dev|stable]], [[DebianPkg:testing/utils/dpkg-dev|testing]], [[DebianPkg:unstable/utils/dpkg-dev|unstable]]
 * Dependencies: DebianPkg:unstable/utils/dpkg-dev
 * Automatic repositories: No
 * Incoming mechanism: No
Line 69: Line 284:
 * GPG signing:
 * A [http://vipie.studentenweb.org/dev/debarchiver/ debarchiver how-to]. An other nice [http://debian.wgdd.de/debhowtos/howto-aptrep.de.html debarchiver how-to (in German)]

=== mini-dinstall ===
 * Goals: Miniature version of dak.
 * Pro: Doesn't require a ["PostgreSQL"] database.
 * Cons:
 * Package:
 * Distributions: Sarge, Sid
 * Dependencies: apt-utils, python2.3, python-apt.
 * GPG signing: No
 * Inclusion of .deb without .changes: Yes
 * Several versions of each package: Yes
 * <<Anchor(mkdebidx)>>[[https://www.mirbsd.org/cvs.cgi/contrib/hosted/tg/deb/mkdebidx.sh|mkdebidx]] is a shell (DebianPkg:mksh) script wrapping dpkg-scanpackages, dpkg-scansources, generating Release and Release.gpg files, and producing a nice XHTML/1.1 index (currently only package-centric view, but dist-/suite-centric views planned) of packages in a full, pinnable, repository with multiple dists and suites (only scales up to a hundred or two packages though; with 904 packages in a repository at the employer, 900 of them in one dist+suite, it takes a while to finish but still works). There may be plans to write a [[DebianPkg:fusionforge|FusionForge]] plugin for repository handling based on this.

<<Anchor(trivialArchive)>>
==== Quick instructions to create a trivial local archive with dpkg-scanpackages ====
We use {{{/var/lib/mydebs}}} as the example location
  * Add this line to {{{/etc/apt/sources.list}}} {{{
  deb file:/var/lib/mydebs ./
  }}}

  * Create the directory {{{
  mkdir -p /var/lib/mydebs
  }}}

  * Every time you put new files in the directory, you'll have to scan the directory and update aptitude: {{{
  ( cd /var/lib/mydebs && dpkg-scanpackages . | gzip > Packages.gz )
  aptitude update
  }}}

  * Now you can install the package normally, using aptitude or apt-get

=== aptly ===
 * Goals: manage local repositories, snapshot them and publish
 * Pros:
    * supports multiple versions of package in one repo
    * has supported for mirroring in the same tool
 * Download & Documentation: http://www.aptly.info/
 * Distributions: [[DebianPkg:stable/utils/aptly|stable]], [[DebianPkg:testing/utils/aptly|testing]], [[DebianPkg:unstable/utils/aptly|unstable]]
 * Source: [[https://github.com/smira/aptly|GitHub]]
 * Automatic repositories: Yes (?)
 * Incoming mechanism: Yes (inoticoming -> aptly repo include)
 * Pools: Yes
 * GPG signing: Yes
 * Inclusion of .deb without .changes: Yes
 * Several versions of each package: Yes
 * HOWTOs:
  * aptly tutorials: https://www.aptly.info/tutorial/
  * Example of setup with incoming: http://oar.imag.fr/wiki:aptly

=== debify ===
 * Goal: takes a directory full of Debian packages and creates a signed and properly-structured Debian repository (wrapper around aptly)
 * Download: https://hub.docker.com/r/spotify/debify/ (docker container)
 * Source: [[https://github.com/spotify/debify|GitHub]]

=== local-apt-repository ===
 * Goal: Create local repository from packages dropped into a certain directory
 * Pros:
   * Simple to use
 * Cons:
   * Repository available only to the local host, not to other machines
 * Distributions: [[DebianPkg:testing/admin/local-apt-repository|testing]], [[DebianPkg:unstable/admin/local-apt-repository|unstable]]
 * Automatic repositories: Yes
 * Incoming mechanism: Not needed. Packages are added by putting them into a fixed directory
 * Pools: No
 * GPG signing: No
 * Inclusion of .deb without .changes: Yes
 * Several version of each pacakge: ?

=== dr ===

 * Goals: managing small Debian repositories
 * Source: [[https://github.com/KanoComputing/kano-repository-manager|GitHub]]
 * Documentation: [[https://github.com/KanoComputing/kano-package-system/wiki|wiki]]
 * Distributions:
Line 83: Line 354:

=== apt-ftparchive ===
 * Goals: Superset of dpkg-scanpackages and dpkg-scansources.
 * Pro: Does not rely on any external programs aside from gzip.
 * Cons:
 * Package:
 * Distributions: Woody, Sarge, Sid
 * Dependencies:
 * Automatic repositories: No
 * Incoming mechanism: No
 * Pools: No
 * GPG signing: No

=== dpkg-scanpackages and dpkg-scansources ===
 * Goals:
 * Pro:
 * Cons:
 * Package:
 * Distributions: Woody, Sarge, Sid
 * Dependencies:
 * Automatic repositories: No
 * Incoming mechanism: No
 * Pools: No
 * GPG signing: No
 * Inclusion of .deb without .changes:
 * Several versions of each package:
 * HOWTO: https://radek.io/2015/06/21/dr-setup/

=== laniakea ===

 * Source: [[https://github.com/lkorigin/laniakea|GitHub]]

=== Pulp ===

Supports various packaging formats via plugins, including one for Debian.

 * Website: https://pulpproject.org/

== Debian Repository Mirroring Tools ==

=== ftpsync ===
 * This is '''THE RECOMMENDED MIRRORING TOOL''' for public mirror sites.
 * Goals: The official archive mirroring tool
  * Instruction: [[http://www.debian.org/mirror/ftpmirror|Setting up a Debian archive mirror]]
  * tarball: http://ftp-master.debian.org/ftpsync.tar.gz
  * git repository: git clone https://ftp-master.debian.org/git/archvsync.git
 * Mirroring with the rsync protocol is strongly recommend.

=== reprepro for partial mirroring ===
 * Description: Debian package repository producer
 * Package: DebianPts:reprepro
 * HOWTOs:
  * [[http://www.infrastructureanywhere.com/documentation/additional/mirrors.html|Creating a Debian/Ubuntu mirror]]
  * [[PartialBackportMirrorWithPackageApproval|Partial backport mirror with package approval with reprepro]]

=== debmirror ===
 * Description: Debian partial mirror script, with ftp and package pool support
 * Package: DebianPts:debmirror

=== apt-mirror ===
 * Description: APT sources mirroring tool
 * Package: DebianPts:apt-mirror

=== debpartial-mirror ===
 * Description: tools to create partial Debian mirrors
 * Package: DebianPts:debpartial-mirror

=== apt-move ===
 * Description: Maintain Debian packages in a package pool
 * Package: DebianPts:apt-move

=== aptly ===
 * Description: Mirror remote repositories and manage local repositories, take snapshots, merge them, pull packages from one snapshot to another, publish snapshots as repositories
 * Download & Documentation: http://www.aptly.info/
 * Distributions: [[DebianPkg:testing/utils/aptly|testing]], [[DebianPkg:unstable/utils/aptly|unstable]]
 * Source: [[https://github.com/smira/aptly|GitHub]]
 * Presentation: [[http://smira.ru/aptly_devops_meetup_eng.pdf|PDF]]

=== anonftpsync (deprecated) ===
 * Description: The previous official mirroring tool
 * Pros:
   * Ease of use and few dependencies,
   * Bash script
 * Cons:
   * Lacks flexibility
   * Does not implement latest (2012 A.D.) mirror features (deprecated)
 * Download: [[http://www.debian.org/mirror/anonftpsync|anonftpsync]]
 * Package: not in Debian
 * Dependencies: bash, rsync

== Related software ==

 * [[DebianPkg:netselect]] selects the fastest mirrors from a list you give, and [[DebianPkg:netselect-apt]] does the same from all existing mirrors.
 * [[DebianPkg:apt-spy]] does something similar with a different method.
 * [[DebianPkg:dput]] and [[DebianPkg:dupload]]: uploads one or more Debian packages into a repository.
 * [[http://elonen.iki.fi/code/parse-apt-files.inc|parse-apt-files.inc]] PHP-script by Jarno Elonen produces a nice [[http://elonen.iki.fi/code/unofficial-debs/|XHTML-summary]] of available packages in a repository - [[http://debian.wgdd.de/debian/parse-apt-files.inc|enhanced version]] for special usage with (but not limited to) [[#debarchiver|debarchiver]]. There seems to be some [[http://potatosalad.gotdns.com/2005/08/01/repository/|efforts]] to develop a wordpress-plugin based on these scripts.

== See also ==

 * DebianPkg:software-properties-gtk
 * SecureApt
 * RepositoryInstructions
 * [[https://www.debian.org/doc/manuals/repository-howto/repository-howto|Debian Repository HOWTO (Obsolete Documentation)]]
 * [[https://debian-administration.org/article/717/Setting_up_a_personal_secure_apt_repository|debian-administration.org - Setting up a personal secure apt repository]] (archived)
 * [[https://www.digitalocean.com/community/tutorials/how-to-use-reprepro-for-a-secure-package-repository-on-ubuntu-14-04|How to Use Reprepro for a Secure Package Repository]] (2014)
 * [[https://wiki.debian.org/Derivatives/CensusFull?action=fullsearch&context=180&value=title%3ADerivatives%2FCensus%2F+regex%3A%5B%5E%23%5D%5Cs%5C*%5CsArchive%5Cstool.%5Cs%5B%5E%2F*%5D&fullsearch=Text|Archive tools used by Debian derivatives]]
Line 109: Line 438:
=== Related software ===
 * netselect selects for you the fastest mirrors among the list you give and netselect-apt does it among all existing mirrors.
 * apt-spy does something similar with a different method.
 * dput uploads one or more Debian packages into a repository.
CategoryPackageManagement | CategoryDeveloper

Translation(s): 한국어(Korean)


This document summarizes the process of setting up a Debian package repository. It does not describe the format of a Debian repository.

Care has been taken to provide the most accurate information at the time of writing. Please fix any identified mistakes.

Debian Repository Types

There are 2 kinds of repositories from user's perspective:

archive style

apt line

apt-pinning

secure APT

status

official archive

"deb http://example.org/debian unstable main"

Yes

Yes

preferred

trivial archive

"deb http://example.org/debian ./"

No

Yes

deprecated

These archives have different meta-data structure. Both archives can store actual package files. Many older repository HOWTOs (e.g. old "Debian Reference (sarge)" and "APT HOWTO (sarge)") address creation of a "trivial archive" and are problematic since the "trivial archive" lacks support for apt-pinning meta-data used by APT Preferences due to the collision of 2 types of Release files.

For the secure APT compatibility, the modern package archive must be signed by GPG.

References:

Repository instructions

The best practices for creating repository usage instructions are available in RepositoryInstructions.

Debian Repository Generation Tools

The full package archive similar to the official archive can be created using:

  • dak
  • mini-dak

The Private Package Archive (PPA) can be created on a web server with a shell account using:

  • reprepro
  • mini-dinstall

people.debian.org is installed with these packages. The PPA archives created on these hosts should only be used for small low-volume experimental archives only.

Do not run high-volume repositories without consulting the host server's maintainer(s).

dak (Debian Archive Kit)

  • Goals: Hosting the official Debian repositories.
  • Pros:
    • It's the official solution
  • Cons:
    • Depends on python and PostgreSQL
    • Lack of documentation
    • Designed for large repositories.
  • Download: git repository.

  • Distributions: not in Debian
  • Dependencies:
    • python
    • postgresql (optional)
  • Automatic repositories: Yes
  • Incoming mechanism: Yes
  • Pools: Yes
  • GPG signing: Yes
  • Inclusion of .deb without .changes: No
  • Several versions of each package: not in the same Packages instance
  • Wiki Page: dak

mini-dak

  • Goals: Hosting new Debian architectures (Partial and lightweight reimplementation of dak in shell script and with no database dependencies)
  • Pros:
    • Easy to setup: edit a config file and run a script to generate the whole structure
    • No database (the pool is the database)
    • All .changes files kept for later possible importing into the master repository
    • Supports mail notifications and does extensive logging
    • Auto package obsoleting
    • Repository snapshotting
    • Supports multiple suites from the Distribution field on the .changes file
    • Additionally supports multipool (splitting each arch into its own pool, to ease partial mirroring)
    • Supports upload ACLs based on gpg public keys
    • Mirror push via ssh
  • Cons:
    • Slow on huge repositories (due to not using a real db mainly)
    • Has been written and tested mainly as a slave archive, so might have some hardcoded stuff which should be fixed to make it work as a master server
    • Still has some quirks to be fixed
  • Download: https://www.hadrons.org/~guillem/debian/mini-dak/

  • Distributions: not in Debian
  • Dependencies: ('grep Requires: *' on the source tree)
    • apt-utils
    • procmail
    • gnupg
    • wget
    • ssh (optional)
    • bzip2 (optional)
    • quinn-diff (optional)
  • Automatic repositories: Yes
  • Incoming mechanism: Yes
  • Pools: Yes
  • GPG signing: Yes
  • Inclusion of .deb without .changes: No
  • Several versions of each package: ??
  • Sites using it:

reprepro

mini-dinstall

debarchiver

  • Goals: Make a simpler version of dak.
  • Pros:
    • Easy to use incoming mechanism - even on remote systems - by using a cron-job
    • Packages can be moved into a distribution by
      1. reading the Distribution value from .changes file or
      2. directly putting the whole package into a distributions-incoming directory.
    • Standard repository (can be pinned)
  • Cons:
    • No Pool-architecture at the moment
    • Some useful checks are missing
    • Cleaning needs to be done manually
  • Package: debarchiver

  • Distributions: oldstable, stable, testing, unstable

  • Dependencies: unstable/devel/debarchiver

    • adduser
    • apt-utils (recommended) | dpkg-dev
    • opalmod (Perl modules)
    • gnupg (optional)
  • Automatic repositories: Yes
  • Incoming mechanism: Yes
  • Pools: No (but suggested somewhere at BTS).

  • GPG signing: Yes (with gnupg, post-Sarge feature).
  • Inclusion of .deb without .changes: ??
  • Several versions of each package: ??
  • HOWTOs:

debpool

  • Goals: Lightweight replacement for dak using a pool layout.
  • Pros:
    • No external dependencies
    • Easy to use incoming mechanism
    • Standard repository (can be pinned)
  • Cons:
    • Not actively maintained since 2008-10-30 (see latest development)

    • no checking of older packages being replaced with new ones
    • no notification of what is going on (no mails when new packages are added)
  • Package: debpool (REMOVED)

  • Distributions: experimental

  • Dependencies:
    • perl
    • gnupg (optional)
  • Automatic repositories: Yes
  • Incoming mechanism: Yes
  • Pools: Yes
  • GPG signing: Yes (with gnupg).
  • Inclusion of .deb without .changes: ??
  • Several versions of each package: ??
  • Wiki page: debpool

DebMarshal

  • Goals: Maintain multiple snapshots from upstream distros, to permit staging.
  • Pros:
    • Fast
    • No database server needed (BerkeleyDB)
  • Cons:
    • Lack of documentation
    • Hasn't been released (No version available, SVN repo has only trunk).
  • Download: http://code.google.com/p/debmarshal/

  • Distributions: not in Debian
  • Automatic repositories: Yes
  • Incoming mechanism: Yes
  • Pools: Yes
  • GPG signing: Yes
  • Inclusion of .deb without .changes: ??
  • Several versions of each package: ??
  • Wiki page: debmartial

  • Presentation: ODP and PDF files

Built by Google for their use.

apt-ftparchive

  • Goals: Superset of dpkg-scanpackages and dpkg-scansources.
  • Pros:
    • Does not rely on any external programs aside from gzip.
    • Creates Release and Contents files without providing *.changes
  • Cons:
    • Can be slow on large repositories, unless the input file (?FileList) is sorted first (the sort command works).

    • ?
  • Package: apt

  • Distributions: oldstable, stable, testing, unstable, experimental

  • Dependencies: unstable/admin/apt-utils

  • Automatic repositories: No (Yes with dupload)
  • Incoming mechanism: No (Yes with custom move cron script with dupload)
  • Pools: Yes
  • GPG signing: No (Yes with dupload with script)
  • Inclusion of .deb without .changes: Yes
  • Several versions of each package: Yes
  • HOWTOs:

Quick instructions to create a trivial local archive with apt-ftparchive

We use /var/lib/mydebs as the example location

  • Create the directory

      mkdir -p /var/lib/mydebs
  • Every time you put new files in the directory, you'll have to scan the directory and update:

      cd /var/lib/mydebs
      apt-ftparchive packages . > Packages
      apt-ftparchive release . > Release
  • Add this line to /etc/apt/sources.list

      deb [allow-insecure=yes] file:/var/lib/mydebs ./
  • Now you can install the package normally. If apt-get asks "Install these packages without verification?", answer "Y" to install. That's because this local repository is not signed.

      apt-get update
      apt-get install mydeb

dpkg-scanpackages and dpkg-scansources

  • Goals: Generate Packages index files
  • Pros:
    • Use only bare bone tool provided by dpkg
    • Creates Release and Contents files without providing *.changes with additional scripts.
  • Cons:
    • Cannot create Release or Contents files by themselves
  • Package: dpkg

  • Distributions: oldstable, stable, testing, unstable

  • Dependencies: unstable/utils/dpkg-dev

  • Automatic repositories: No
  • Incoming mechanism: No
  • Pools: No
  • GPG signing: No
  • Inclusion of .deb without .changes: Yes
  • Several versions of each package: Yes
  • mkdebidx is a shell (mksh) script wrapping dpkg-scanpackages, dpkg-scansources, generating Release and Release.gpg files, and producing a nice XHTML/1.1 index (currently only package-centric view, but dist-/suite-centric views planned) of packages in a full, pinnable, repository with multiple dists and suites (only scales up to a hundred or two packages though; with 904 packages in a repository at the employer, 900 of them in one dist+suite, it takes a while to finish but still works). There may be plans to write a FusionForge plugin for repository handling based on this.

Quick instructions to create a trivial local archive with dpkg-scanpackages

We use /var/lib/mydebs as the example location

  • Add this line to /etc/apt/sources.list

      deb file:/var/lib/mydebs ./
  • Create the directory

      mkdir -p /var/lib/mydebs
  • Every time you put new files in the directory, you'll have to scan the directory and update aptitude:

      ( cd /var/lib/mydebs && dpkg-scanpackages . | gzip > Packages.gz )
      aptitude update
  • Now you can install the package normally, using aptitude or apt-get

aptly

  • Goals: manage local repositories, snapshot them and publish
  • Pros:
    • supports multiple versions of package in one repo
    • has supported for mirroring in the same tool
  • Download & Documentation: http://www.aptly.info/

  • Distributions: stable, testing, unstable

  • Source: GitHub

  • Automatic repositories: Yes (?)
  • Incoming mechanism: Yes (inoticoming -> aptly repo include)

  • Pools: Yes
  • GPG signing: Yes
  • Inclusion of .deb without .changes: Yes
  • Several versions of each package: Yes
  • HOWTOs:

debify

local-apt-repository

  • Goal: Create local repository from packages dropped into a certain directory
  • Pros:
    • Simple to use
  • Cons:
    • Repository available only to the local host, not to other machines
  • Distributions: testing, unstable

  • Automatic repositories: Yes
  • Incoming mechanism: Not needed. Packages are added by putting them into a fixed directory
  • Pools: No
  • GPG signing: No
  • Inclusion of .deb without .changes: Yes
  • Several version of each pacakge: ?

dr

  • Goals: managing small Debian repositories
  • Source: GitHub

  • Documentation: wiki

  • Distributions:
  • Automatic repositories:
  • Incoming mechanism:
  • Pools:
  • GPG signing:
  • Inclusion of .deb without .changes:
  • Several versions of each package:
  • HOWTO: https://radek.io/2015/06/21/dr-setup/

laniakea

Pulp

Supports various packaging formats via plugins, including one for Debian.

Debian Repository Mirroring Tools

ftpsync

reprepro for partial mirroring

debmirror

  • Description: Debian partial mirror script, with ftp and package pool support
  • Package: debmirror

apt-mirror

  • Description: APT sources mirroring tool
  • Package: apt-mirror

debpartial-mirror

apt-move

  • Description: Maintain Debian packages in a package pool
  • Package: apt-move

aptly

  • Description: Mirror remote repositories and manage local repositories, take snapshots, merge them, pull packages from one snapshot to another, publish snapshots as repositories
  • Download & Documentation: http://www.aptly.info/

  • Distributions: testing, unstable

  • Source: GitHub

  • Presentation: PDF

anonftpsync (deprecated)

  • Description: The previous official mirroring tool
  • Pros:
    • Ease of use and few dependencies,
    • Bash script
  • Cons:
    • Lacks flexibility
    • Does not implement latest (2012 A.D.) mirror features (deprecated)
  • Download: anonftpsync

  • Package: not in Debian
  • Dependencies: bash, rsync

See also


CategoryPackageManagement | CategoryDeveloper