Preparation of a store-backed Amazon EC2 image based on Debian Squeeze

************ THIS IS A DRAFT ********* ************ IT WILL CHANGE A LOT ********* ************ PLEASE DON'T USE IT RIGHT AWAY *********

This page describes how to create a Debian image for Amazon EC2 cloud computing.

your PC:

cd $HOME
wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip 
unzip ec2-api-tools.zip
unzip ec2-ami-tools.zip
mkdir ec2
rsync -ar ec2-api-tools-*/* ec2
rsync -ar ec2-ami-tools-*/* ec2
rm -rf ec2-api-tools*
rm -rf ec2-ami-tools*

your PC:

export EC2_HOME=$HOME/ec2
export PATH=$PATH:$EC2_HOME/bin
export JAVA_HOME=/usr

your PC:

export AMI=ami-XXXXXXX
export TYPE=m1.large
export KEY=YOUR_KEY
export EC2_PRIVATE_KEY=/mnt/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem 
export EC2_CERT=/mnt/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem 

your PC:

ec2-run-instances $AMI -k $KEY -t $TYPE
watch -n5 ec2-describe-instances
export user=root
export host=ec2-50-16-180-121.compute-1.amazonaws.com

rsync                       \
--rsh="ssh -i ${KEY}.pem"   \
--rsync-path="sudo rsync"   \
{cert,pk}-*.pem             \
${user}@${host}:/mnt/

ssh -i ${KEY}.pem ${user}@${host}

instance:

export SUDO=sudo

instance:

$SUDO aptitude update

instance:

$SUDO aptitude install unzip

Those packages will be needed later

instance:

$SUDO aptitude install ruby libopenssl-ruby openjdk-6-jre-headless
$SUDO aptitude install debootstrap

Download the 'Amazon EC2 API Tools' and 'Amazon EC2 AMI Tools', copy the content into one directory

instance:

cd $HOME
wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip 
unzip ec2-api-tools.zip
unzip ec2-ami-tools.zip
mkdir ec2
rsync -ar ec2-api-tools-*/* ec2
rsync -ar ec2-ami-tools-*/* ec2
rm -rf ec2-api-tools*
rm -rf ec2-ami-tools*

and export the EC2_HOME environment variable to this directory.

instance:

export EC2_HOME=$HOME/ec2
export PATH=$PATH:$EC2_HOME/bin

instance:

export JAVA_HOME=/usr

Export some account related environment variables

instance:

export EC2_PRIVATE_KEY=/mnt/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem 
export EC2_CERT=/mnt/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem 
export AWS_USER_ID=120000000012
export AWS_ACCESS_KEY_ID=20XXXXXXXXXXXXXXXX20
export AWS_SECRET_ACCESS_KEY=45fn4kHFssGFHnjfnkFHdsfsd9855454refeGFD

instance:

export PROJECT_NAME=bionode
export PROJECT_VERSION=alpha4

instance:

export IMG=${PROJECT_NAME}-${PROJECT_VERSION}.img

instance:

export ARCH=amd64
or
export ARCH=i386

instance:

dd if=/dev/zero of=$IMG bs=4M count=1024
 1024+0 records in
 1024+0 records out
 1073741824 bytes (1.1 GB) copied, 35.1134 s, 30.6 MB/s

instance:

mkfs.ext3 -F $IMG
 mke2fs 1.41.12 (17-May-2010)
 Filesystem label=
 OS type: Linux
 Block size=4096 (log=2)
 Fragment size=4096 (log=2)
 Stride=0 blocks, Stripe width=0 blocks
 65536 inodes, 262144 blocks
 13107 blocks (5.00%) reserved for the super user
 First data block=0
 Maximum filesystem blocks=268435456
 8 block groups
 32768 blocks per group, 32768 fragments per group
 8192 inodes per group
 Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376
 
 Writing inode tables: done                            
 Creating journal (8192 blocks): done
 Writing superblocks and filesystem accounting information: done
 
 This filesystem will be automatically checked every 36 mounts or
 180 days, whichever comes first.  Use tune2fs -c or -i to override.

instance:

$SUDO mkdir /mnt/ec2

instance:

$SUDO mount -o loop $IMG /mnt/ec2

Choose a Debian mirror close to Amazon's location

instance:

$SUDO debootstrap --arch $ARCH squeeze /mnt/ec2 http://ftp.debian.org
or
$SUDO debootstrap --arch $ARCH squeeze /mnt/ec2 http://mirror.cc.columbia.edu/debian/
 I: Retrieving Release
 I: Retrieving Packages
 I: Validating Packages
 I: Resolving dependencies of required packages...
 I: Resolving dependencies of base packages...
 I: Found additional required dependencies: insserv libbz2-1.0 libdb4.8 libslang2 
 I: Found additional base dependencies: libnfnetlink0 libsqlite3-0 
 I: Checking component main on http://ftp.debian.org...
 I: Retrieving libacl1
 I: Validating libacl1
 I: Retrieving adduser
 I: Validating adduser
 
 [...]
 
 I: Configuring aptitude...
 I: Configuring tasksel-data...
 I: Configuring tasksel...
 I: Base system installed successfully.

instance:

$SUDO chroot /mnt/ec2

chroot:

mount -t proc none /proc
mount -t devpts none /dev/pts

chroot:

aptitude update

Choose e.g. en_US.UTF-8

chroot:

aptitude install locales
dpkg-reconfigure locales

chroot:

aptitude install curl
aptitude install ssh
/etc/init.d/ssh stop
aptitude install makedev
ln -s /sbin/MAKEDEV /dev
cd /dev

the following takes a minute

chroot:

for dev in "zero null console generic"; do MAKEDEV $dev; done

chroot:

aptitude purge isc-dhcp-client isc-dhcp-common dhcp3-client
aptitude install dhcpcd

Install Packages

chroot:

aptitude install med-imaging 

Add your own repositories

chroot:

cat << EOLIST | tee -a /etc/apt/sources.list
deb     http://www.YOURDEBIANREPO.com/debian ./
deb-src http://www.YOURDEBIANREPO.com/debian ./
EOLIST
aptitude update
aptitude install YOURPACKAGE

It is better to deactivate the password permanently, but for now change the password

chroot:

  /# passwd
  Enter new UNIX password: 
  Retype new UNIX password: 
  passwd: password updated successfully

Edit /etc/fstab

chroot:

 proc          /proc    proc    defaults     0     0
 /dev/sda1     /        ext3    defaults     0     0

Edit /etc/network/interfaces

chroot:

 auto lo
 iface lo inet loopback
 auto eth0
 iface eth0 inet dhcp

Remove the standard hostname

chroot:

rm -f /etc/hostname

chroot:

update-rc.d -f hwclock.sh remove
update-rc.d -f hwclockfirst.sh remove
umount /proc
umount /dev/pts

chroot:

aptitude clean

Exit from your chroot environment

chroot:

exit

$SUDO cp -r /lib/modules/$(uname -r) /mnt/ec2/lib/modules/

instance:

$SUDO umount -l /mnt/ec2/

instance:

ec2-bundle-image -i $IMG -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $AWS_USER_ID
 Please specify a value for arch [i386]: 
 Bundling image file...
 Splitting /tmp/bionode.img.tar.gz.enc...
 Created bionode.img.part.00
 Created bionode.img.part.01
 Created bionode.img.part.02
 Created bionode.img.part.03
 Created bionode.img.part.04
 Created bionode.img.part.05
 Created bionode.img.part.06
 Created bionode.img.part.07
 Created bionode.img.part.08
 Created bionode.img.part.09
 Created bionode.img.part.10
 Created bionode.img.part.11
 Created bionode.img.part.12
 Created bionode.img.part.13
 Created bionode.img.part.14
 Created bionode.img.part.15
 Created bionode.img.part.16
 Generating digests for each part...
 Digests generated.
 Creating bundle manifest...
 ec2-bundle-image complete.

If a error happens you have to create the bucket inside the management console first

instance:

ec2-upload-bundle -b ${PROJECT_NAME} -m /tmp/$IMG.manifest.xml -a $AWS_ACCESS_KEY_ID -s $AWS_SECRET_ACCESS_KEY
 Creating bucket...
 Uploading bundled image parts to the S3 bucket bionode-alpha1 ...
 Uploaded bionode.img.part.00
 Uploaded bionode.img.part.01
 Uploaded bionode.img.part.02
 Uploaded bionode.img.part.03
 Uploaded bionode.img.part.04
 Uploaded bionode.img.part.05
 Uploaded bionode.img.part.06
 Uploaded bionode.img.part.07
 Uploaded bionode.img.part.08
 Uploaded bionode.img.part.09
 Uploaded bionode.img.part.10
 Uploaded bionode.img.part.11
 Uploaded bionode.img.part.12
 Uploaded bionode.img.part.13
 Uploaded bionode.img.part.14
 Uploaded bionode.img.part.15
 Uploaded bionode.img.part.16
 Uploading manifest ...
 Uploaded manifest.
 Bundle upload completed.

Print aki instance:

curl http://instance-data/latest/meta-data/kernel-id ; echo

instance:

export AKI=aki-74f4051d

instance:

ec2-register ${PROJECT_NAME}/$IMG.manifest.xml -n bionode-alpha2  -K $EC2_PRIVATE_KEY -C $EC2_CERT --kernel="$AKI"
 IMAGE ami-7eee1e17

Terminate your instance if no longer needed

your PC:

ec2-terminate-instances i-XXXXXXXX

See also

References