Differences between revisions 17 and 18
Revision 17 as of 2009-10-22 19:35:33
Size: 10789
Comment: abstractions
Revision 18 as of 2009-10-22 19:52:46
Size: 6200
Comment: Moved subsection to separate page.
Deletions are marked like this. Additions are marked like this.
Line 101: Line 101:
The motivation behind this effort is to have the USB stick function like a regular hard drive and provide the basic functionality for some virtual or real machine to do exactly what it is supposed to do (and nothing much else). Again, a compressed file system would be lovely, but a plain journaled ext2 seems to allow a good start.

The installation is performed via deboostrap. cdebootstrap should also work, but it has not for me. Then to be added is a bootloader. I tried with grub2, but it reads from hd0 instead of hd1 when tested on a Windows machine, and I failed to change that. If hd0 is correct for a diskless client I cannot tell. I was then successful with extlinux, an ext2-compatible syslinux. The following worked for me.

Some parameters to be used throughout the remainder of these instructions

{{{
   device=/dev/sdb # other systems may present the stick as sda
   partno=1 # number of partition you want to boot from
   mountpoint="/mnt/stick" # directory for the stick to show up
   mirror="http://ftp.de.debian.org/debian" # from where to download
   # packages that shall be installed
   packages="boinc-client autodock autogrid autodocktools gromacs r-cran-qtl r-recommended"
}}}

Now the real thing starts

{{{
   sudo fdisk $device # create a partition
   mke2fs -j $device$partno # journaled ext2 file system
   tune2fs -c 0 $device$partno # don't check
   [ -d "$mountpoint" ] || sudo mkdir "$mountpoint" # prepare mount point
   mount -t ext2 $device$partno "$mountpoint"
   debootstrap --include=openssh-client,nfs-common squeeze "$mountpoint" $mirror
}}}

A disadvantage of this setup is the limited space to cache the packages - the binaries eat up the disk space that is needed to unpack them. And apt-get does not care about removing packages from /var/cache/apt to free disk space during installation. After the debootstrap, one should install in fractions rather than all in one - also depending on the size of your medium. A 4GB USB stick was found to be sufficient, but far less comfy than originally anticipated. Go for at least 8GB.

{{{
   chroot /mnt/stick
   # some basic infrastructure
   apt-get install debian-keyring build-essential dhcp3-client
   # debian-med
   apt-get install med-bio-dev
   apt-get clean
   # more advanced packages - adjust to your liking
   for p in $packages; do echo "Installing $p"; apt-get --no-install-recommends install $p && apt-get clean; done
}}}

While still in the chroot, the password of root needs to be set.
{{{
   passwd root
}}}

And how to access the network should be specified - please adjust:
{{{
   cat > /etc/network/interfaces <<EONET
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp
EONET
}}}

For getting the beast to boot, I attempted the following. The kernel should be installed, first, and the installation inspects the /proc directory, so this is prepared for it, too.

{{{

   #sudo chroot /mnt/stick # entering the chroot just created, if not already in

   cat /etc/fstab # make sure you really are in the chroot and
                  # hence want to overwrite this
   echo "Will be overwritten in ten seconds if you don't CTRL-C"
   sleep 10 # you shall not copy and paste blindly
   cat > /etc/fstab << EOFSTAB
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
/dev/sdb1 / ext3 defaults,errors=remount-ro 0 1
EOFSTAB
   mount /proc
   apt-get install linux-image
   # answering 'Y' to the questions asked, except for the
   # preparation for the initrd, which is fine - press "don't cancel"
}}}

Now the boot loader should be installed. It was not clear from the documentation ([[http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=548424|#548424]]) about how the configuration file should be named - so some good soul please improve the description given here.

{{{
   # still in the chroot
   cat > /extlinux.conf <<EOCFG
DEFAULT debian-med-2.6.30
LABEL debian-med-2.6.30
SAY Debian Med from USB stick
KERNEL /boot/vmlinuz-2.6.30-1-amd64
APPEND ro root=/dev/sdb1 initrd=/boot/initrd.img-2.6.30-1-amd64
EOCFG
   exit # now back from the chroot

   # here I have indeed done the cat mbr.bin > /dev/sdb , but
   # it should also work without it - I think. If not, see
   # http://syslinux.zytor.com/wiki/index.php/EXTLINUX

   sudo apt-get install syslinux
   extlinux -i /mnt/stick/boot

   sync # does not help too much with journaled file systems, though

   chroot /mnt/stick umount /proc
   umount /mnt/stick
}}}

References:

 * http://syslinux.zytor.com/wiki/index.php/SYSLINUX
 * http://syslinux.zytor.com/wiki/index.php/EXTLINUX
A very neat (my favorite) setup for a machine to be booted with a USB stick is with the stick acting like a regular drive. Detailed instructions, almost an executable script, are given on what has not become this sub-page: [[DebianMed/LiveCD/StickPlain]].

Well, the term "LiveCD" is somewhat historic. But we all know that we basically mean a

Preinstalled portable Debian Med software stack

The motivation for this effort is to bring the packages offered in Debian even more closely towards a direct usability. This may be via your USB stick, a local/remote virtual image or just a better collaboration via a completely homogeneous infrastructure.

This page was put up to discuss pros and cons for various ways to prepare such a LiveCD. Points for discussion are

  • selection of packages
    • the adaptation of those selection for the various media
    • the influence that this selection may have on the description of tasks in Debian Med
  • extra efforts that may be beyond main Debian
    • integration of public data
    • integration of educational material
  • most suitable tool to prepare the LiveCD
  • who does what, what folks external to the core of Debian can we get involved

Selections for Scenarios

All selections shall be additive.

'Scenario'

'Selection'

'Comment'

development happens on machine

build-essential|med-bio-dev

machine is worked with locally

med-bio

features many programs with GUI

machine is for computations only

'app list'

GUI-free sub-task is being discussed for Debian Med

executed on cloud

euca2ools

a major motivation behind this effort

integration of data

getData

the integration work will look very differently for cloud and non-cloud installs, getData is still in development and yet not prepared at all for the demands of the cloud

strong interest in special packages

install those packages

and consider mailing the list about it

netboot

extension of regular LiveCDs

some advanced packages of general interest

debfoster, deborphan

a ready live CD does not profit from these, but its production does.

Preparation

There shall be two tools proposed

  • live-helper
  • plain chroot plus boot loader
  • vmbuilder

Protocol for Live Helper

Live Helper, see DebianLive for details, was prepared for preparing the LiveCD in a multi-stage manner. One does a configuration first, a script 'lh_config' prepares a good (and for us now until we want images and other bits added at various places sufficient) template for that. A second step will involve the actual compilation of the thus specified infrastructure.

The following sniplet will build an image to dd to the USB stick. Please follow the instructions on ?DebianLive/Howto/USB for further insights.

sudo apt-get install live-helper
lh_config -b usb-hdd \
  --distribution squeeze --categories "main contrib non-free" \
  -p standard --packages '^(libcv-dev|autodock|gromacs|ballview|autodocktools|dropbear|boinc-client|r-cran-qtl|r-recommended)$' \
  -m http://ftp2.de.debian.org/debian/
sudo lh_build

The complete build time - over DSL and with a now elderly laptop is about 45 minutes. The resulting hard drive image can then be dumped to the USB stick's device (not its partition). With a disk partitioning tool the remainder of the stick may be rendered usable again - the DD also writes the partition table and this needs to be adapted for.

dd if=binary.img of=/dev/sdb
fdisk /dev/sdb
...

As a hard disk image, the live CD can also be run in a variety of virtual setups. The expert tool for such though is "vmbuilder", described below.

When preparing this live medium, one should be aware that the device is not directly mountable, but that on the FAT or FAT32 filesystem a very large file is created, which represents the real image. I have failed to make changes to the local setup, e.g. /etc/hosts changed or so, persistent. This is certainly possible, somehow, but this overview would need a good soul to describe how. I want the medium to be the only medium that the machine sees, e.g. to boot from otherwise diskless clients. If someone would know how for instance to create a second partition on the USB stick (ok, I have done that) and use that together with the aufs/squashfs to achieve persistency within the USB stick, not with the (non-existent) hard disk.

One persistency is achievable, and the image possibly compressed, this setup would seem ideal for many causes.

Protocol for netboot

this section is currently under preparation - don't try it yet, but feel free to correct if you know better

This scenario prepares one single root image that is served via NFS. Changes are written to a copy-on-write (COW) image, which is different for every machine. Everything is prepared in complete analogy to the regular Live CD that is prepared with live-helper:

sudo apt-get install live-helper

For an easier copy and paste by avoiding line breaks

server="pc07.inb.uni-luebeck.de" # place your server here
packages="libcv-dev|autodock|gromacs|ballview|autodocktools"
packages="$packages|lvm2|openssh-server|openssh-client"
packages="$packages|boinc-client|r-cran-qtl|r-recommended"
packages="$packages|build-essential|nfs-common"

lh_config -b net \
  --net-root-server $server --net-cow-server $server \
  --distribution squeeze --categories "main contrib non-free" \
  -p standard \
  --packages "^($packages)\$" \
  -m http://ftp2.de.debian.org/debian/ \
  --chroot-filesystem ext3
sudo lh_build

lvm2 was added to increase flexibility with locally added disks, should the need arise. We had issues with a squashfs-modules packages that was suddenly demanded from us and we did not have it nor could easily build it. The "--cache-stages disabled" should be possible to omit.

The Wiki page Network_Image_Server has more information on this matter. It also gives a reference for a tutorial for Creating_a_Test_environment .

Protocol for plain chroot plus boot loader

A very neat (my favorite) setup for a machine to be booted with a USB stick is with the stick acting like a regular drive. Detailed instructions, almost an executable script, are given on what has not become this sub-page: DebianMed/LiveCD/StickPlain.

Protocol for vmbuilder

To be written - David?