Differences between revisions 10 and 11
Revision 10 as of 2004-12-26 01:35:58
Size: 16758
Editor: anonymous
Comment:
Revision 11 as of 2004-12-26 01:42:43
Size: 16803
Editor: anonymous
Comment:
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
The path to creating the procedure began when information on how create a debian partial mirror without having a full mirror in the first place couldn't be found. deb-mirror and friends all required that one start with a full mirror which isn't an option for for everyone. The decision was made to try creating a custom partial mirror using a network install cd and what was downloaded using aptitude (apt-get with a nice frontend) on an older system that was being refurbished. It took more work than expected, but it now works. This article was begun it was found that information on how create a debian partial mirror without having a full mirror in the first place did not appear to be available. deb-mirror and friends all required that one start with a full mirror which doesn't work if you don't have dsl, or have bandwidth caps. The decision was made to try creating a custom partial mirror using a network install cd and what was downloaded using aptitude (apt-get with a nice frontend) on an older system that was being refurbished. It took more work than expected, but it now functional.
Line 340: Line 340:
=== 3. Get overrides.<dist>.* === === 3. Get overrides.&ltdist&gt.* ===
Line 344: Line 344:
If you want to includes packages from non-US as well you ought to create another indices directory (e.g. indices-non-US), and get the overrides from a non-US mirror as well (from /debian-non-US/indices-non-US). If you want to includes packages from non-US you ought to create another indices directory (e.g. {{{indices-non-US}}}), and get the overrides from a non-US mirror (from {{{/debian-non-US/indices-non-US}}}).
Line 399: Line 399:
     * {{{OverrideDir}}} is the directory which contains your overrides ''Note that this means that if you want non-US, you need another {{{.conf}}} with your non-US directories in it.''      * {{{OverrideDir}}} is the directory which contains your overrides ''Note that this means that if you want non-US, you need another {{{.conf}}} with your non-US directories.''
Line 402: Line 402:
     * The path in quotes after {{{BinDirectory}}} (and before the curly brace), is the location, relative to {{{ArchiveDir}}} of the binary packages (well actually the alphabetic directories, under which are the package directories).
     * Under {{{BinDirectory}}}, {{{Packages}}} is the full pathname of the Packages (without .gz even if you are creating compressed files) file you are creating (in this case the debian-installer Packages.gz).
     * The path in quotes after {{{BinDirectory}}} (and before the curly brace), is the location, relative to {{{ArchiveDir}}} of the binary packages (well actually the alphabetic directories, which have the package directories under them).
     * Under {{{BinDirectory}}}, {{{Packages}}} is the full pathname of the Packages (without .gz even if you are creating compressed files) file you are creating (in this case the debian-installer {{{Packages.gz}}}).

Building a Custom Debian CD Set

2004-11-22 DanielDickinson

Introduction

This page will describe the steps used to create a custom debian install cd using a partial mirror mostly gathered from a prototype system's apt cache. The intructions here assume i386 but should work for any architecture simply by replacing i386 in the instructions with the architecture of your choice.

This article was begun it was found that information on how create a debian partial mirror without having a full mirror in the first place did not appear to be available. deb-mirror and friends all required that one start with a full mirror which doesn't work if you don't have dsl, or have bandwidth caps. The decision was made to try creating a custom partial mirror using a network install cd and what was downloaded using aptitude (apt-get with a nice frontend) on an older system that was being refurbished. It took more work than expected, but it now functional.

Quick Start

  • Download a DebianInstaller network install cd (netinst.iso).

  • Install Debian on a prototype system (this is optional)

  • ?AptGet packages (or aptitude if you prefer, as I do) -> hint: you don't have to install, just download to the apt cache, as long apt-move works.

  • AptMove to a directory of your choice.

  • create a pool directory on the filesystem where you will build your new cd.

  • copy the pool directory from network install cd to the directory you just created.
  • copy from the directory you apt-move'd to to the pool directory (you could just do this first, but if you realize you missed something the first time around, you'll need to do this).

  • get installer-i386 from a debian mirror which is from the same release of DebianInstaller as your network install cd (unless you also build the installer and associated packages). get overrides.<dist>.''' files from http://<mirror>/debian/indices

  • place in your debian-dir (i.e. the same directory as the pool directory you created above).
  • gunzip the overrides in-place the indices directory (you should probably keep the original gzipped versions too)

  • generate packages files.
    • generate DebianInstaller packages files:

      • DebianInstaller lives in <debian-dir>''dists''<dist>/main/debian-installer/binary-i386

      • so create the appropriate subdirectories (e.g. mkdir -p /debian/dists/sarge/main/debian-installer/binary-i386).

      • in a directory for your scripts and config files for this project place the following apt.conf:

            APT {
                  ["FTPArchive"] {
                        Release {
                                Origin "debian-cd";
                                Label "yousarge";
                                Suite "testing";
                                Version "0.1";
                                Codename "sarge";
                                Architectures "i386";
                                Components "main contrib";
                                Description "Your Sarge CD Set";
                        };
                  };
            };
      • And the following yourcdname-di.conf (creating the directories described therein, as necessary):

            Dir {
                ArchiveDir "/debian";
                OverrideDir "indices";
                CacheDir "/tmp";
            };

    TreeDefault {
        Directory "pool/";
    };

    BinDirectory "pool/main" {
        Packages "dists/sarge/main/debian-installer/binary-i386/Packages";
        BinOverride "override.sarge.main";
         ExtraOverride "override.sarge.extra.main";
    };

    Default {
        Packages {
                Extensions ".udeb";
                Compress ". gzip";
        };

        Contents {
                Compress "gzip";
        };
    };
  • execute the command

        apt-ftparchive -c apt.conf generate yourcdname-di.conf
  • repeat for binary-i386:
    • using the following yourcdname.conf instead of yourcdname-di.conf:

          Dir {
              ArchiveDir "/debian";
              OverrideDir "indices";
              CacheDir "/tmp";
          };

    TreeDefault {
        Directory "pool/";
    };

    BinDirectory "pool/main" {
        Packages "dists/sarge/main/binary-i386/Packages";
        BinOverride "override.sarge.main";
         ExtraOverride "override.sarge.extra.main";
    };

    BinDirectory "pool/contrib" {
        Packages "dists/sarge/contrib/binary-i386/Packages";
        BinOverride "override.sarge.contrib";
         ExtraOverride "override.sarge.extra.contrib";
    };

    BinDirectory "pool/non-free" {
        Packages "dists/sarge/non-free/binary-i386/Packages";
        BinOverride "override.sarge.non-free";
        ExtraOverride "override.sarge.extra.non-free";
    };

    Default {
        Packages {
                Extensions ".deb .udeb";
                Compress ". gzip";
        };

        Contents {
                Compress "gzip";
        };
    };
  • create a list of packages you want on the cd (probably just a list of files in your pool dir).
  • Assuming you have debian-cd installed, follow the directions in /usr/share/debian-cd.
  • In the mirrorcheck step, if any missing Depends: are listed, download the packages into your pool dir to correct that, regenerate your packages files, and start the debian-cd instructions over again (you may need to exit the shell you started the procedure in because debian-cd alters the environment while it builds the cd ["set"]).
  • In the make list step wach for debconf_required errors, and as above download the needed packages, regenerate package lists, and restart the debian-cd procedure.
  • assuming sucessful completion of the debian-cd cd-building procedure burn your cd and test.

Detailed Instructions

1. Getting Your Packages

The Hard Way

  • Figure out what packages you'll need (this is the hard part because of dependencies).
  • Download them individually (if you make a list and use wget on the list, it wouldn't be as painful, but keeping up with changing versions would rapidly become painful) into the pool directory in the filesystem where you'll be doing your work (e.g. /home/debian/pool).

  • Don't forget any dependencies (by hand! ugh!).

The Easy Way, Part I: Get Your Base System

  1. Download a ?DebiainInstaller network install cd.

  2. Use the install cd and install your base system on some computer.
  3. Copy the network install cd to the filesystem where you'll be doing your work (e.g. /home/debian).

The Easy Way, Part II: Getting Additional Packages

  1. Use aptitude or ?AptGet to download the packages you want on the system and cd.

  2. Get AptMove (e.g. apt-get apt-move)

  3. Modify apt-move.conf to use your pool directory as its local mirror (LOCALDIR)

  4. AptMove your other packages to your pool directory (e.g. /home/debian/pool) by typing: apt-move update

NB: I haven't actually tried using AptMove directly onto the partial mirror because of the way in which I learned what I needed to do, but it should work.

Notes on the use of AptMove

You will need to change apt-move.conf to point to your mirror. See AptMove for details (or man apt-move).

The Slightly Harder Way, Part II: Getting Additional Packages

  1. Use Aptitude or ?AptGet to download the packages you want on the system and cd.

  2. Get AptMove (e.g. apt-get apt-move)

  3. Modify apt-move.conf to use a temporary directory as its local mirror (LOCALDIR)

  4. AptMove your other packages to a temporary directory (e.g. /home/debian/tmp) by typing: apt-move update

  5. Use the following script to copy packages from the temporary directory to your pool directory (e.g. /home/debian/pool). This is my preferred mode of operation because I have more control over what happens to what apt-move calls the 'mirror' (the pool directory), especially when building cd's for testing or unstable, which change quickly.

 #!/bin/bash

 function print_usage {
    echo 
    echo "copy-new-debs-to-pool [--move] [--keep-old] dir-to-copy add-to-dir"
    echo
    echo "paths must be absolute"
    echo "and must point to the directory in which the pool subdirectory resides"
    echo "The pool directory must have main contrib and non-free subdirectories"
    echo
 }

 if [ "$1" = "--move" ] ; then
     MOVE=1
     if [ "$2" = "--keep-old" ] ; then
        KEEP=1
     else
        KEEP=0
     fi
 else
     MOVE=0
     if [ "$1" = "--keep-old" ] ; then
        KEEP=1
     else
        KEEP=0
     fi
 fi

 if [ -z "$1" ]; then 
     print_usage
     exit
 fi

 if [ -z "$2" ]; then
     print_usage
     exit
 fi

 if [ $MOVE -eq 1 -a -z "$3" ]; then
     print_usage
     exit
 else 
     if [ $KEEP -eq 1 -a -z "$3" ]; then
        print_usage
        exit
     fi
 fi

 if [ $MOVE -eq 1 -a $KEEP -eq 1 -a -z "$4" ]; then
     print_usage
     exit
 fi

 if [ $MOVE -eq 0  -a $KEEP -eq 0 ]; then
     ["RC1DIR"]=$1
     ["RC2DIR"]=$2
 else 
     if [ $MOVE -eq 1 -a $KEEP -eq 1 ]; then
        ["RC1DIR"]=$3
        ["RC2DIR"]=$4    
     else
        ["RC1DIR"]=$2
        ["RC2DIR"]=$3
     fi
 fi

 echo "Copying from $["RC1DIR"] to $["RC2DIR"]"

 TMPFILE=`tempfile`

 cd $["RC1DIR"]
 find pool -type f || sort >$TMPFILE
 rc1base=$["RC1DIR"]
 rc2base=$["RC2DIR"]
 cd -
 for rc1file in `cat $TMPFILE`; do
     pkgfullname=`basename $rc1file`
     pkgname=`echo $pkgfullname || cut -f1 -d_`
     rc1ver=`echo $pkgfullname || cut -f2 -d_`
     pooldir=`dirname $rc1file`
     pkgend=`echo $pkgfullname || cut -f3 -d_`
     rc2file=`ls $rc2base/$pooldir/$pkgname\__*\__$pkgend 2>/dev/null`
     if [ "$rc2file" ]; then
        rc2maxver="!!!!!!!!!!!!!!!!!"
        rc2maxfile="!!!!!!!!!!!!!!!!"
        for rc2match in $rc2file; do
            rc2pkgfullname=`basename $rc2match`
            rc2ver=`echo $rc2pkgfullname || cut -f2 -d_`
            if [ "$rc2ver" ">" "$rc2maxver" ]; 
                then rc2maxver=$rc2ver
                rc2maxfile=$rc2match
            fi
        done
        rc2file=$rc2maxfile
        if [ $rc1ver '>' $rc2maxver ] ; then
            echo "Copying $rc1file newer than $rc2file"
            if [ $MOVE -eq 1 ] ; then
                mv -f $rc1base/$rc1file `dirname $rc2base/$rc1file`
                if [ $KEEP -eq 1 ]; then
                    echo "Not  removing obsolete $rc2file"
                else
                    echo "Removing obsolete $rc2file"
                    rm -f $rc2base/$rc2file
                fi
            else
                cp $rc1base/$rc1file `dirname $rc1base/$rc1file`
                if [ $KEEP -eq 1 ]; then
                    echo "Not removing obsolete $rc2file"
                else
                    rm -f $rc2base/$rc2file
                fi
            fi
            continue
        fi
     else
        echo "Copying $pkgname not in $["RC2DIR"]"
        install -d $["RC2DIR"]/`dirname $rc1file`
        if [ $MOVE -eq 1 ]; then
            mv -f $["RC1DIR"]/$rc1file `dirname $["RC2DIR"]/$rc1file`
        else
            cp $["RC1DIR"]/$rc1file `dirname $["RC2DIR"]/$rc1file`      
        fi
     fi
 done

 rm -f $TMPFILE

Alternate Method

Instead of copying the network install cd first, copy your Apt cache to your desired pool directory first (as above), then use the above script to copy from the network install pool directory to your pool directory.

2. Get DebianInstaller and udebs

Option 1

Should only be used when using a stable distribution's network install cd and stable packages, for sarge or later because the udebs must match the installer for a successful install. If you insist on using it with testing or unstable, make sure you're using the right versions of the udebs, especially kernel, for the installer. You have been warned.

  1. Download installer-i386 from a ["Debian"] mirror (e.g. ftp://ftp.debian.org/debian/dists/sarge/main/current)

  2. You should have the correct udebs in your tree if you have downloaded the installer-i386 from the same distribution as your network install cd

Option 2

  1. Build DebianInstaller and udebs. (See ?DebianInstallerBuild).

  2. Copy udebs to your pool directory, using something like the following script:

    IFS=$'\n'
    pushd /debian-installer/installer/build/localudebs
    for file in `ls`; 
            do pkg=`echo $file || cut -f1 -d_`
            firstletter=`expr substr $pkg 1 1`
            if [ $firstletter = l ] 
                    then libletters=`expr substr $pkg 1 3`
                    if [ $libletters = "lib" ]; then
                            dirletters=`expr substr $pkg 1 4`
                            mkdir -p /debian/pool/main/$dirletters/$pkg
                            cp $file /debian/pool/main/$dirletters/$pkg
                    else
                            mkdir -p /debian/pool/main/$firstletter/$pkg
                            cp $file /debian/pool/main/$firstletter/$pkg
                    fi
            else
                    mkdir -p /debian/pool/main/$firstletter/$pkg
                    cp $file /debian/pool/main/$firstletter/$pkg
            fi
    done
  
    popd

You obviously need to change the paths. This script is not really finished, but it's good enough for now.

3. Get overrides.&ltdist&gt.*

  1. Download /debian/indices/overrides.<dists>.'''.gz for your distro (e.g. for sarge, /debian/indices/overrides.sarge.'''.gz) into the directory indices which is in the same root directory as your pool (mirror) directory (e.g. /debian/pool, so /debian/indices).

  2. Gunzip the overrides, keeping the compressed files.

If you want to includes packages from non-US you ought to create another indices directory (e.g. indices-non-US), and get the overrides from a non-US mirror (from /debian-non-US/indices-non-US).

4. Generate Packages.gz and Release files for your 'distribution'

  • generate DebianInstaller packages files:

    • DebianInstaller lives in <debian-dir>''dists''<dist>/main/debian-installer/binary-i386

    • so create the appropriate subdirectories (e.g. mkdir -p /debian/dists/sarge/main/debian-installer/binary-i386).

    • in a directory (e.g. custom-cd-scripts) for your scripts and config files for this project place the following apt.conf:

          APT {
                ["FTPArchive"] {
                      Release {
                              Origin "debian-cd";
                              Label "yousarge";
                              Suite "testing";
                              Version "0.1";
                              Codename "sarge";
                              Architectures "i386";
                              Components "main contrib";
                              Description "Your Sarge CD Set";
                      };
                };
          };
    • And the following customcd-di.conf (creating the directories described therein, as necessary):

          Dir {
              ArchiveDir "/debian";
              OverrideDir "indices";
              CacheDir "/tmp";
          };

    TreeDefault {
        Directory "pool/";
    };

    BinDirectory "pool/main" {
        Packages "dists/sarge/main/debian-installer/binary-i386/Packages";
        BinOverride "override.sarge.main";
         ExtraOverride "override.sarge.extra.main";
    };

    Default {
        Packages {
                Extensions ".udeb";
                Compress ". gzip";
        };

        Contents {
                Compress "gzip";
        };
    };
  • ArchiveDir is the parent directory of your pool and dists subdirectories

  • OverrideDir is the directory which contains your overrides Note that this means that if you want non-US, you need another .conf with your non-US directories.

  • CacheDir is your temporary (cache) directory.

  • Under TreeDefault, Directory indicates the location of your pool (archive) directory

  • The path in quotes after BinDirectory (and before the curly brace), is the location, relative to ArchiveDir of the binary packages (well actually the alphabetic directories, which have the package directories under them).

  • Under BinDirectory, Packages is the full pathname of the Packages (without .gz even if you are creating compressed files) file you are creating (in this case the debian-installer Packages.gz).

  • execute the command

        apt-ftparchive -c apt.conf generate yourcdname-di.conf
  • repeat for the binary-i386 (main) directory:

    • using the following customcd.conf instead of customcd-di.conf:

          Dir {
              ArchiveDir "/debian";
              OverrideDir "indices";
              CacheDir "/tmp";
          };

    TreeDefault {
        Directory "pool/";
    };

    BinDirectory "pool/main" {
        Packages "dists/sarge/main/binary-i386/Packages";
        BinOverride "override.sarge.main";
         ExtraOverride "override.sarge.extra.main";
    };

    BinDirectory "pool/contrib" {
        Packages "dists/sarge/contrib/binary-i386/Packages";
        BinOverride "override.sarge.contrib";
         ExtraOverride "override.sarge.extra.contrib";
    };

    BinDirectory "pool/non-free" {
        Packages "dists/sarge/non-free/binary-i386/Packages";
        BinOverride "override.sarge.non-free";
        ExtraOverride "override.sarge.extra.non-free";
    };

    Default {
        Packages {
                Extensions ".deb .udeb";
                Compress ". gzip";
        };

        Contents {
                Compress "gzip";
        };
    };
  • If you are using any non-US packages you will need to repeat the above Packages file creation in each of the appropriate subdirectories.

... to be continued ...