DEBPARTIAL-MIRROR IDEAS

This page is to coordinate debpartial-mirror development.

.. debpartial-mirror: http://packages.debian.org/unstable/net/debpartial-mirror

Introduction

debpartial-mirror is a tool to create a local partial mirror of apt-repository.

The partial-mirror can be used both as a source of packages to build *Custom Debian Distributions* archives, cd-images or local APT repositories.

The packages included into the partial mirror can be downloaded from remote APT repository or local directories.

The list of packages that are included into the partial-mirror is specified by means of a configuration file: it is also possible to filter from a package list.

Mirror organization

The partial mirror is organized as a collection of Backends. A Backend is a repository with packages from one source (a local or a remote repository):

  ''partial_mirror''
    sarge/
    sid/
    debian-np/
    local-np/
    br/
    it/
    ....

In turn each backend, has a fixed structure:

  /partial_mirror/backend/
    dists/
    pool/

the dists/ directory contains the Packages.gz, Sources.gz and Release files for the Backend, while the pool/ directory contais an apt pool-style tree of packages.

Merged backends

Once each Backend is filled, it is possible to create several *Merged Backends*.

A Merged Backend is a Backend built using a list of Backends from the local mirror that contains a dists/ dir that merges the contents of the different dists/ subdirectories of the selected Backends and a pool/ subdirectory built using symbolic (or hard) links to the packages included on the merged Backends.

In addition, when the Merged Backend is created, it is possible to filter which packages to include from the selected Backends.

Backend snapshots

Each backend has a list of binary and source packages inside the dists/ subdirectory and a copy of the corresponding files into the pool/ dir.

The standard way of updating each backend would be to download the new Packages and Sources lists, remove the old files from the pool and download the new ones.

A very interesting option could be to support multiple snapshots of the same backend, the idea would be to keep diferent versions of the dists/ subdirectory using a file structure like the following:

  /partial_mirror/backend/
    snapshots/YYYYMMDD/dists/
    snapshots/YYYYMMDD+1/dists/
    ...
    dists/
    pool/

With this schema each dists/ subdirectory contains the corresponding lists of source and binary packages and the pool/ subdir contains the source and package files of all included snapshots.

Of course the system has to allow the selective removal of one snapshot (i.e., remove the 20041109 snapshot), that is, it has to support the removal of the source and package files that belong to the selected snapshot and are not included on any of the rest of snapshots.

Shared pool

Extending the idea of the snapshots presented before we could use a shared pool/ to store the source and package files included on each backend.

This could reduce the download time and the pool size; if a file is included into two backends it is only installed once.

The problem with this model is similar to the one described earlier: to remove unwanted files from previous backends without breaking all the rest we have to calculate the differences as described when talking about removing a snapshot.

Program description

.. Note: This is different from the actual implementartion, the following

• could be used as trace for the man-page.

SYNOPSIS

  debpartial-mirror [-c=file]
    {update [backend, [....]] || merge [backend, [....]] ||
    || check || clean [backend, [....]]}

DESCRIPTION

debpartial-mirror will accept some command in the way apt-get does, and read conf information from a configuration file (default /etc/debpartial-mirror/debpartial-mirror.conf) that in turn can include other conf files.

update

  this command let debpartial-mirror to update the backends that
  compose the partial mirror. If no backend is specified, all
  backends will be update. Else, only the listed backends will be
  updated.

merge

  this command will create one (or more) merge backend. The list
  of backends to be used to merge is specified in the main
  conf-file.

After update and/or merge, debpartial-mirror will clean up the mess removing old and uneeded files from it's structure.

Notes about the source code

The backend class

Properties:

        need_update (boolean)

Members:

        Dists -> A class that realize backend/dists
        PackageList -> A class that contains the list of packages that

        need_update (boolean)

Members:

        Dists -> A class that realize backend/dists
        PackageList -> A class that contains the list of packages that
                       should be copied into the partial-mirror of this
                       backend

Methods:

        create->
        update -> Sync the partial-mirror with the sources repositories
        __create_package_list -> Generate a package list reading from
                                 main conf-file.

The dists class

Properties:

        got_files

Methods:

        update -> Update the Dists files (Packages, Release and Sources)
        _get -> Get each file