Differences between revisions 70 and 78 (spanning 8 versions)
Revision 70 as of 2017-11-10 12:33:09
Size: 16290
Comment: Move install-debian-installer script before the command using it.
Revision 78 as of 2020-06-08 17:48:50
Size: 16755
Comment: Update note about the status of this project
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:
Work in progress, thus the instructions below are not guaranteed to work at the moment. {{{#!wiki caution
'''Please note:''' This project is not actively being worked on, and is not supported by the cloud team. The instructions below are not guaranteed to work at the moment, but may be interesting as a starting point for future work.
}}}
Line 76: Line 78:
apt update
apt install --yes parted
Line 84: Line 88:
DI_VERSION=20150422+deb8u4 DI_VERSION=current
Line 118: Line 122:
Start the instance with an extra volume of 1 GiB, which will persist after termination ({{{/dev/sdb=:1:false}}}). Pass the script that will format the volume and download Debian-Installer (see below). Start the instance with an extra volume of 1 GiB, which will persist after termination ({{{/dev/sdb=:1:false}}}). Pass the script that will format the volume and download Debian-Installer (see below).  You may need to set the option {{{--subnet-id}}} if your EC2 account is old.
Line 158: Line 162:
Note that AMI names can not contain the {{{+}}} character. Note that AMI names can not contain the {{{+}}} character.  Correct the name
and description according to the version of the installer that you used.
Line 162: Line 167:
  --name debian-jessie-installer-amd64-20150422-deb8u4 \
  --description 'Debian-Installer 8 (jessie) version 20150422+deb8u4 for amd64' \
  --name debian-stretch-installer-amd64-20170615-deb9u2 \
  --description 'Debian-Installer 9 (Stretch) version 20170615+deb9u2/ for amd64' \
Line 215: Line 220:
This time, pass the pressed file as user data. This time, pass the [[#Example_preseed_file|preseed]] file as user data.
Line 452: Line 457:
== See also ==

 * [[Cloud/AmazonEC2Image]]
 * [[Cloud]].
Line 453: Line 463:
See also [[Cloud/AmazonEC2Image]], and [[Cloud]].
CategoryVirtualization | CategorySystemAdministration

Debian-Installer AMIs

Amazon Machine Images for the DebianInstaller

Please note: This project is not actively being worked on, and is not supported by the cloud team. The instructions below are not guaranteed to work at the moment, but may be interesting as a starting point for future work.

This page is made of two parts:

  • AMI build script: commands to start a "helper" instance that will download a copy of Debian-Installer on a specific partition, and turn this partition into a machine image.

  • Debian installation: commands to start a Debian-Installer image that will install Debian on a specific parition, and turn this partition into a machine image.

Both parts follow similar principles, therefore some commands are put in common as shell functions.

The operations in the cloud are controlled from the local computer using programs from the awscli package. Therefore, the user's credentials never leave the local computer.

AMI build script

The procedure below runs a helper instance with an attached volume of 1 GiB, partitions it, downwoads Debian-Installer on it and sets up GRUB2 to boot the installer. The snapshot of the resulting volume is registered as a machine image.

First, declare a few accessory functions:

runHelperInstance takes an image ID as a first argument, and appends the following arguments to a run-instances command of the aws cli. By default it also sets the instance shutdown behaviour to terminate, uses a t2.nano instance with a 1 GiB block on /dev/sdb and expects a script file in ./install-debian-installer. This can be overridden with environment variables, as seen in the source code below:

runHelperInstance() {
  IMAGE_ID=$1
  shift
  aws ec2 run-instances  \
    --image-id $IMAGE_ID \
    --instance-initiated-shutdown-behavior ${INSTANCE_BEHAVIOR-terminate} \
    --instance-type ${INSTANCE_TYPE-t2.nano} \
    --block-device-mappings ${INSTANCE_BLKD-"[{\"DeviceName\":\"/dev/sdb\",\"Ebs\":{\"DeleteOnTermination\":false,\"VolumeSize\":1}}]"} \
    --user-data ${INSTANCE_DATA-file://install-debian-installer} \
    --query Instances[0].InstanceId \
    --output text \
    $*
}

The next functions should be self-explanatory.

getVolumeIdentifier() {
  aws ec2 describe-instances \
    --instance-id $1 \
    --query 'Reservations[0].Instances[0].BlockDeviceMappings[?DeviceName==`/dev/sdb`].Ebs[].VolumeId' \
    --output text
}

waitForInstanceStatus() {
  while [ ! $( aws ec2 describe-instances \
                 --instance-id $1 \
                 --query Reservations[0].Instances[0].State.Name \
                 --output text | tee /dev/stderr) = $2 ]                                                
    do sleep 30
  done
}

getInstanceIp() {
  aws ec2 describe-instances \
    --instance-id $1 \
    --query 'Reservations[0].Instances[0].NetworkInterfaces[0].Association.PublicIp' \
    --output text
}

Copy the following script under the name install-debian-installer.

   1 #!/bin/sh -ex
   2 # License: CC0
   3 
   4 apt update
   5 apt install --yes parted
   6 parted -s /dev/xvdb mklabel msdos mkpart primary 0% 100%
   7 mke2fs -L debian-installer /dev/xvdb1 -F
   8 mount LABEL=debian-installer /mnt/
   9 
  10 cd /mnt
  11 
  12 ARCH=amd64
  13 DIST=stable
  14 DI_VERSION=current
  15 MIRROR=http://cloudfront.debian.net
  16 BASEURL=$MIRROR/debian/dists/$DIST/main/installer-$ARCH/$DI_VERSION/images/netboot/xen
  17 
  18 wget $BASEURL/initrd.gz $BASEURL/vmlinuz
  19 
  20 mkdir -p boot/grub
  21 
  22 cat > boot/grub/grub.cfg <<__END__
  23 set default='0'
  24 set timeout='3'
  25 menuentry 'Debian Installer ($DI_VERSION $ARCH)' {
  26 insmod part_msdos
  27 insmod ext2
  28 set root='(hd0,msdos1)'
  29 linux /vmlinuz root=LABEL=debian-installer ro console=tty1 console=ttyS0 auto=true priority=critical url=http://169.254.169.254/latest/user-data DEBIAN_FRONTEND=text
  30 initrd /initrd.gz
  31 }
  32 __END__
  33 
  34 grub-install /dev/xvdb --boot-directory=/mnt/boot
  35 
  36 cd /
  37 umount /mnt
  38 
  39 halt

Then, configure a Debian image ID.

HELPER_IMAGE=ami-b6b568d0 # Debian Stretch (Asia North-East 1)

Start the instance with an extra volume of 1 GiB, which will persist after termination (/dev/sdb=:1:false). Pass the script that will format the volume and download Debian-Installer (see below). You may need to set the option --subnet-id if your EC2 account is old.

HELPER_INSTANCE=$(runHelperInstance $HELPER_IMAGE | tee /dev/stderr)

Get the volume's identifier.

TARGET_VOLUME=$(getVolumeIdentifier $HELPER_INSTANCE | tee /dev/stderr)

Wait that the scripts shuts down the instance.

waitForInstanceStatus $HELPER_INSTANCE terminated

Snapshot the volume and register it as a machine image.

TARGET_SNAPSHOT=$( aws ec2 create-snapshot \
  --volume-id $TARGET_VOLUME \
  --query SnapshotId \
  --output text | tee /dev/stderr)

while aws ec2 describe-snapshots --snapshot-ids $TARGET_SNAPSHOT --query Snapshots[0].State --output text  |
  tee /dev/stderr |
  grep -q pending
do
  sleep 10
done

Delete the snapshoted volume, not needed anymore.

aws ec2 delete-volume --volume-id $TARGET_VOLUME

Note that AMI names can not contain the + character. Correct the name and description according to the version of the installer that you used.

aws ec2 register-image \
  --name debian-stretch-installer-amd64-20170615-deb9u2 \
  --description 'Debian-Installer 9 (Stretch) version 20170615+deb9u2/ for amd64' \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs": {"SnapshotId":"'$TARGET_SNAPSHOT'"}}]' \
  --architecture x86_64 \
  --root-device-name /dev/sda1 \
  --virtualization-type hvm

Images

This is run in the Asia North-East zone from the Debian AMI Account (account number 379101102735).

Work in progress; may be deleted.

379101102735/debian-wheezy-installer-amd64-20130613_deb7u1

Coming soon !

ap-northeast-1

ami-a1861ca0

ap-southeast-1

ami-e03164b2

ap-southeast-2

ami-fbb529c1

eu-west-1

ami-efb85b98

us-east-1

ami-29fea140

us-west-1

ami-0eaa9d4b

us-west-2

ami-1ece562e

sa-east-1

ami-bfb81ea2

Replicated and made public by hand using the web console.

Debian installation

Instance a Debian-Installer image and pass it a preseed file via the user data.

Note the similarity between the commands below and the commands above. They probably can be factorised in a single script that takes appropriate parameters.

Chose an installer image (for instance here: 379101102735/debian-wheezy-installer-amd64-20130613_deb7u1 on ap-northeast-1).

INSTALLER_IMAGE=ami-b38e14b2

If you want an image of 8 GiB:

INSTANCE_BLKD='[{"DeviceName":"/dev/sdb","Ebs":{"DeleteOnTermination":false,"VolumeSize":8}}]'

If you use the network console, pass the name of your key (in this example, name).

SSH_KEY_NAME="name"

This time, pass the preseed file as user data.

INSTANCE_DATA=file://preseed.txt

Run the instance in a security group that has the SSH port open.

SECURITY_GROUP="Your security group's ID" # At this stage, the instance does not need to be accessible.

INSTALLER_INSTANCE=$(runHelperInstance $INSTALLER_IMAGE \
                       --key-name $SSH_KEY_NAME \
                       --security-group-id $SECURITY_GROUP | tee /dev/stderr)

Wait that the instance is running.

waitForInstanceStatus $INSTALLER_INSTANCE running

Get the volume's identifier.

TARGET_VOLUME=$(getVolumeIdentifier $INSTALLER_INSTANCE | tee /dev/stderr)

The rest is work in progress: preseed correctly to have accurate menu.lst and fstab files, prevent the creation of a swap partition, install cloud-init from backports …

TARGET_SNAPSHOT=$( aws ec2 create-snapshot \
  --volume-id $TARGET_VOLUME \
  --query SnapshotId \
  --output text | tee /dev/stderr)

while aws ec2 describe-snapshots --snapshot-ids $TARGET_SNAPSHOT --query Snapshots[0].State --output text  |
  tee /dev/stderr |
  grep -q pending
do
  sleep 10
done

Delete the snapshoted volume, not needed anymore.

aws ec2 delete-volume --volume-id $TARGET_VOLUME

Note that AMI names can not contain the + character.

aws ec2 register-image \
  --name MyDebianSystem \
  --description 'Custom Debian system created with a custom preseed file' \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs": {"SnapshotId":"'$TARGET_SNAPSHOT'"}}]' \
  --architecture x86_64 \
  --root-device-name /dev/sda1 \
  --virtualization-type hvm

Example preseed file

#### Adapted from http://www.debian.org/releases/wheezy/example-preseed.txt
### Localization
# Preseeding only locale sets language, country and locale.
d-i debian-installer/locale string en_US

# Keyboard selection.
d-i keyboard-configuration/xkb-keymap select us

# netcfg will choose an interface that has link if possible. This makes it
# skip displaying a list if there is more than one interface.
d-i netcfg/choose_interface select auto

# Any hostname and domain names assigned from dhcp take precedence over
# values set here. However, setting the values still prevents the questions
# from being shown, even if values come from dhcp.
d-i netcfg/get_hostname string unassigned-hostname
d-i netcfg/get_domain string unassigned-domain

### Network console
# Use the following settings if you wish to make use of the network-console
# component for remote installation over SSH. This only makes sense if you
# intend to perform the remainder of the installation manually.
d-i anna/choose_modules string network-console
d-i network-console/authorized_keys_url string http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key

### Using the CloudFront mirror (local in the EC2).
d-i mirror/country string manual
d-i mirror/http/hostname string cloudfront.debian.net
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string

# Suite to install.
#d-i mirror/suite string testing
# Suite to use for loading installer components (optional).
#d-i mirror/udeb/suite string testing

### Account setup
# Skip creation of a root account (use sudo with "administrator" account).
d-i passwd/root-login boolean false
# Create an "administrator" account
d-i passwd/user-fullname string administrator
d-i passwd/username string administrator
# FIXME: no password should be set after cloud-init is installed
d-i passwd/user-password password FIXME
d-i passwd/user-password-again password FIXME

### Clock and time zone setup
# Controls whether or not the hardware clock is set to UTC.
d-i clock-setup/utc boolean true

# You may set this to any valid setting for $TZ; see the contents of
# /usr/share/zoneinfo/ for valid values.
d-i time/zone string US/Eastern

# Controls whether to use NTP to set the clock during the install
d-i clock-setup/ntp boolean true

### Partitioning
d-i partman-auto/disk string /dev/xvdb
# - regular: use the usual partition types for your architecture
d-i partman-auto/method string regular
# - atomic: all files in one partition
d-i partman-auto/choose_recipe select atomic
# This makes partman automatically partition without confirmation, provided
# that you told it what to do using one of the methods above.
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

## Controlling how partitions are mounted
# Mount by label as UUIDs will change.
d-i partman/mount_style select label

### Base system installation
# The kernel image (meta) package to be installed; "none" can be used if no
# kernel is to be installed.
#d-i base-installer/kernel/image string linux-image-486

### Apt setup
# You can choose to install non-free and contrib software.
#d-i apt-setup/non-free boolean true
#d-i apt-setup/contrib boolean true
# Select which update services to use; define the mirrors to be used.
# Values shown below are the normal defaults.
#d-i apt-setup/services-select multiselect security, updates
#d-i apt-setup/security_host string security.debian.org

# Additional repositories, local[0-9] available
#d-i apt-setup/local0/repository string \
#       http://local.server/debian stable main
#d-i apt-setup/local0/comment string local server
# Enable deb-src lines
#d-i apt-setup/local0/source boolean true
# URL to the public key of the local repository; you must provide a key or
# apt will complain about the unauthenticated repository and so the
# sources.list line will be left commented out
#d-i apt-setup/local0/key string http://local.server/key

# By default the installer requires that repositories be authenticated
# using a known gpg key. This setting can be used to disable that
# authentication. Warning: Insecure, not recommended.
#d-i debian-installer/allow_unauthenticated boolean true

### Package selection
#tasksel tasksel/first multiselect standard, web-server
# If the desktop task is selected, install the kde and xfce desktops
# instead of the default gnome desktop.
#tasksel tasksel/desktop multiselect kde, xfce

# Individual additional packages to install
#d-i pkgsel/include string openssh-server build-essential
# Whether to upgrade packages after debootstrap.
# Allowed values: none, safe-upgrade, full-upgrade
#d-i pkgsel/upgrade select none

# Some versions of the installer can report back on what software you have
# installed, and what software you use. The default is not to report back,
# but sending reports helps the project determine what software is most
# popular and include it on CDs.
#popularity-contest popularity-contest/participate boolean false

### Finishing up the installation
# Avoid that last message about the install being complete.
d-i finish-install/reboot_in_progress note

# This will prevent the installer from ejecting the CD during the reboot,
# which is useful in some situations.
#d-i cdrom-detect/eject boolean false

# This is how to make the installer shutdown when finished, but not
# reboot into the installed system.
d-i debian-installer/exit/halt boolean true
# This will power off the machine instead of just halting it.
d-i debian-installer/exit/poweroff boolean true

### Preseeding other packages
# Depending on what software you choose to install, or if things go wrong
# during the installation process, it's possible that other questions may
# be asked. You can preseed those too, of course. To get a list of every
# possible question that could be asked during an install, do an
# installation, and then run these commands:
#   debconf-get-selections --installer > file
#   debconf-get-selections >> file

#### Advanced options
### Running custom commands during the installation
# d-i preseeding is inherently not secure. Nothing in the installer checks
# for attempts at buffer overflows or other exploits of the values of a
# preconfiguration file like this one. Only use preconfiguration files from
# trusted locations! To drive that home, and because it's generally useful,
# here's a way to run any shell command you'd like inside the installer,
# automatically.

# This first command is run as early as possible, just after
# preseeding is read.
#d-i preseed/early_command string anna-install some-udeb
# This command is run immediately before the partitioner starts. It may be
# useful to apply dynamic partitioner preseeding that depends on the state
# of the disks (which may not be visible when preseed/early_command runs).
#d-i partman/early_command \
#       string debconf-set partman-auto/disk "$(list-devices disk | head -n1)"
# This command is run just before the install finishes, but when there is
# still a usable /target directory. You can chroot to /target and use it
# directly, or use the apt-install and in-target commands to easily install
# packages and run commands in the target system.
#d-i preseed/late_command string apt-install zsh; in-target chsh -s /bin/zsh

See also


CategoryVirtualization | CategorySystemAdministration